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1. Introduction 

This paper is a sequel to [GGP], where we considered several restriction problems 
in the representation theory of classical groups over local and global fields. Assuming 
the Langlands-Vogan parameterization of irreducible representations, we formulated 
precise conjectures for the solutions of these restriction problems. In the local case, 
our conjectural answer is given in terms of Langlands parameters and certain natural 
symplectic root numbers associated to them. In the global case, the conjectural answer 
is expressed in terms of the central critical value or derivative of a global L-function. 
For the precise statements of the restriction problems and our conjectures, we refer the 
reader to [GGP]. 



The conjectures for the case of special orthogonal groups were contained in the earlier 
papers [GP1] and [GP2] and were suggested by the results of Waldspurger [Wal,2,3], 
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Tunnell-Saito [Tu], [Sa], and Prasad [PI, 2, 3] in certain low rank cases. Since then, 
there have been further results in the orthogonal case, both locally and globally; see, 
for example [P4], [GR], [GJS2], and [PT]. Most notably, in two recent preprints [Wa4] 
and [Wa5] , Waldspurger has made spectacular progress towards to a full resolution of 
the local conjectures of [GP1, GP2]. 

In this paper, we provide some evidence for the conjectures of [GGP] in the unitary 
case. More precisely, we shall consider the restriction problems in the following cases: 

(i) the depth zero supercuspidal L-packets of DeBacker-Reeder[ DR], which are as- 
sociated to tame regular discrete L-parameters; 

(ii) certain low rank cases, such as U(l) x U(l), U(l) x U(2), U(2) x U(2) and U(2) x 
U(3). 

In each case, we shall establish [GGP, Conjecture 16.3]. 



Acknowledgments: W. T. Gan is partially supported by NSF grant DMS-0801071. 
B. H. Gross is partially supported by NSF grant DMS 0901102. D. Prasad was partially 
supported by a Clay Math Institute fellowship during the course of this work. We also 
thank P. Deligne, S. Kudla, G. Lusztig, M. Reeder, D. Rohrlich, and J.-L. Waldspurger 
for their help. 

2. Discrete series parameters 

We begin with the computation of the distinguished character in [GGP, Conjecture 
16.3] 

X = Xn x Xm ■ A M x A N ^< ±1 >, 

which is defined using local root numbers, for some discrete series parameters of the 
unitary groups G = XJ(W) x U(Wo), associated to the quadratic extension of local 
fields k/ko. 

In general, these discrete series parameters have the form 

M = 0M, 

i 

N = 0iV, 

3 

where the are distinct conjugate-symplectic representations and the Nj are distinct 
conjugate-orthogonal representations of the Weil-Deligne group of k. The dimension 
of M is even and the dimension of N is odd. In this case, the centralizer Cm x Cn of 
the Langlands parameter is finite. 
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We will only consider the case here where each Mj = M(ai) and each Nj = N(/3j) 
is one dimensional. Then a, is a character of k x /Nk x with eti\ k x = Wk/k , and f3j is a 
character oi k x /k^ . In this case, we have the component groups 

A M = 0Z/2Z-e, 

A N = 0Z/2Z-/,, 

These vector spaces have dimension equal to dimM and dimiV over Z/2Z, which is as 
large as possible. We have 

M ei= ~ l = M{pn) 

Fix a nontrivial additive character tpo of k which is trivial on ko- By the definition 
of the character x, we have the formulae 

X(ei) = e(M(ai)®N,4> ) 

X (fj) = e{M®N{(3 j )M. 

Using the additivity of the local epsilon factors, this becomes 

k 

xUj) = n e ( afc ^'^°)- 

k 

Since the products 01$ j are all conjugate-symplectic characters of k x , we need a 
formula to compute their root numbers. We will do this is two different cases - when 
k/ko = C/R, which we take up now, and then when k/k is unramified which we do in 
the next section. 

Proposition 2.1. Assume that ko = K and choose an isomorphism z : k — > C. Let 

a = z~ 2a ■ (zz) a = {z/z) a 
be a conjugate-symplectic character of k x , where a is a half integer, and let 

^ = e 2TiTr(i 2 ) = e 2-n(z-z) 

Then 

-1 if a > 0; 



e(a,V>o) = 

Proof. Tate [T, (3.2.5)] gives the formula 



-li/o< 0. 



e(a, ip) = i 



2a 
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when a > and ip — e 2ntTr( - z \ Since ipo(z) = ip(iz), we find 

e(a,ip ) = i 2a ■ = +1 

in this case. When a < we must conjugate the isomorphism z : k — > C to use Tate's 
formula. This changes the character ip , and hence the sign of e. □ 



Corollary 2.2. Assume that k = R 7 choose an isomorphism z : k — > C, and Zei 

■00 = e 27r ^ -z ). // M zs £/ie sum o/ £/ie distinct symplectic characters cti = (z/z) ai , 
where each a ZiaZ/ integer, and N is the sum of the distinct orthogonal characters 
[3j = (z/z) bj , where each bj is an integer, then 

x(ei) = (-ir 

x(fs) = (-i) nj 

where 

m i = #{ r : a i + b r < 0} 

rij = #{r : a r + bj < 0}. 



Finally, we note that in the case when k = R, we may order the distinct characters 
Oij and Pj in the parameter so that 

a± > a 2 > a 3 - ■ ■ in |Z — Z 
h > b 2 > b 5 ■ ■ ■ in Z. 



Corollary 2.3. For i < j 7 we Ziave 

xfe)x(e,) = 

x(/0x(/i) = (— l)™ 43 - 

m «i = #{ r '■ a i + b r > > a_j + & r } 
rijj = #{r : 6j + a r > > 5j + a r }. 



Since we know how to describe the representations in the L-packets of discrete series 
parameters when fc = R [GR], the calculation of x( e i)x( e j) an d x{fi)x{fj) allows us to 
say something about the representation 7r = n(ip, x) — tti ® 7t ' 2 of G(R) with d(ir) = 1. 
The irreducible representations 7Ti and 7r 2 are discrete series representations of even 
and odd dimensional unitary groups, with infinitesimal characters 

ai > a 2 > a% > ■ ■ ■ 
6i > b 2 > h > ■ ■ ■ 

in X* + p respectively. Moreover, in the chambers defined by their Harish- Chandra 
parameters, the simple root walls corresponding to 
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e» - e i+ i is compact -<=^ x( e i) ' x( e i+i) = -1 
fi - f i+ i is compact • x(fi+i) = ~ l - 

More generally, for i < j the positive root 

ti — tj is compact •<=>- x( e i) ' x( e j) — (— 1) J+,J 
/i - fj is compact • x(fj) = 

This determines the signature of the unitary group G(M), and in almost all cases the 
discrete series representation ir. 

We end this section with a remark about branching from U(n, 1) to U(n). According 
to a theorem of Harish-Chandra, an irreducible admissible (q, i^)-module is determined 
by the action of U(g) K , where U(g) denotes the universal enveloping algebra of G, on 
a given K-type which appears in the representation space. Further, the action of 
K x U(g) on the corresponding isotypical component is irreducible. By a theorem of 
Kostant, for 

G = U(n,l) and K = U(n) x U(l), 

U (g) K is generated by the centers of the universal enveloping algebras of G and K, and 
thus is, in particular, abelian. This proves that any irreducible representation of U(n) 
appears with multiplicity at most one in any irreducible representation of U(n, 1). 

3. Depth zero supercuspidals 

In this section, we test the restriction conjecture for some tamely ramified discrete 
parameters (p of unitary groups. We begin by calculating the local root numbers, 
assuming that is non-archimedean and k is the unramified quadratic extension of 
k . 

Proposition 3.1. Assume that k is non-archimedean, and let k be the unramified 
quadratic field extension of k . Let ip be an additive character of k which is trivial on 
both ko and the maximal ideal of the ring of integers Ak, but is nontrivial on A^. Let 
a be a conjugate- symplectic character of k x of conductor f (a). Then 

e(a,ifo) = (-l)'W + i. 

Proof. When k/ko is unramified, every conjugate-symplectic character a has the form 

a — (5 • /i, 

where (3 : k x /k£ — > C x is a conjugate-orthogonal character and n is the unramified 
quadratic character (which is conjugate-symplectic). By [GGP, Section 5] and [FQ, 
Theorem 3], we have 

e(/3,Vo) = +1 
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for any character ipo of k which is trivial on ko- Since \i is unramified, we have [T, (3.4.6)] 

e (a,^o) = e(A^o)-/i(vr /(/3)+n{ ^ ) ). 
Since f(/3) = f(a) and n(ip ) = — 1, this gives the formula in the proposition. □ 

Corollary 3.2. Assume that k is non-archimedean, and let k be the unramified qua- 
dratic field extension of ho. Let ip be an additive character of k which is trivial on both 
k and the maximal ideal of the ring of integers A^, but is nontrivial on A^. Let 

M = QiCii and N = ®jf3j 

where thecti 's are mutually distinct, tamely ramified, conjugate- symplectic characters, 
and the (3j 's are mutually distinct, tamely ramified, conjugate-orthogonal characters. 
Order these characters so that 

ocif3i = a 2 (32 = ■ ■■ = a p (3 p = fx, 

for p > and no other products ctifij = /i. Then 

-1 when i < p; 
+1 when i > p. 



Similarly, 

X(fj) = 

Finally, X (-l, 1) = X (l, -1) = (-l)f. 



— 1 when j < p; 
+1 when j > p. 



Proof. Since our characters are all tamely ramified, we find 

ffaPj) = 1, 

unless % = j < p, in which case the product is equal to the unramified character /i and 
f(cti(3i) = 0. Taking the product of epsilon factors giving x gives the desired result. □ 

We take the parameter 

M = 0M(a } ) 

given by the sum of distinct conjugate-symplectic and distinct conjugate-orthogonal 
characters of k x . We will assume that all of these characters are tamely ramified: 

/K) = /(&) = i. 

The L-packet 11^ of depth zero supercuspidal representations of the pure inner forms 
G = XJ(W) x U(W ) has been constructed by DeBacker and Reeder [DR], and we briefly 
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summarize their results in this case. Let V be a hermitian space of dimension n over k. 
A parameter ip of the above type for the unitary group U(V) = U n gives, by restriction 
to the units of k x , a regular complex character p of the anisotropic torus S = (Ui) n . 
Each embedding 

f:S^XJ(V) 

corresponds to a decomposition of the space V into the direct sum of orthogonal lines, 
stable under the action of S. Write this decomposition, choosing a basis for each line, 
as 

v = Q)kv t . 

The U(V)-conjugacy class of the embedding / depends only on the signs 

tl = (_ 1 )ord(^^) ) 

which must satisfy the one relation 

ord(discV) 

Since the two hermitian spaces V and V of dimension n have distinct hermitian dis- 
criminants, there are exactly 2 n conjugacy classes of embeddings / of S into U(V) and 
U(V'). These conjugacy classes correspond bijectively to the characters x — Xf °f the 
group Ay, where x{ e i) = e i- 

For each embedding / : S — > U(V), there is a unique maximal compact subgroup 
Kf C U(V) which contains the image. This is the subgroup stabilizing the lattice, 

L f = (J) A k Vi, 

where we normalize the basis vectors of our S'-stable lines to satisfy < ord(t> Vi) < 1. 
The reduction mod n of Kf has reductive quotient 

K f ^ U p x U n _ p , 

where p is the number of Vi with (— iy rd ( v i^) — _ i. Hence Kf will be hyperspecial if 
and only if all of the inner products (vi,Vi) have valuations of the same parity. 

The torus S(q) embeds in Kf(q), and the regular tame character p of S(q) allows us 
to construct an irreducible, supercuspidal representation Rf(S,p) of the finite group 
Kf(q), using the method of Deligne and Lusztig. We view this as a representation of 
the compact group Kf, and define the representation 

n x = 7r f , ofU(V) 

as the compact induction of Rf(S,p). These are the 2 n depth zero supercuspidal 
representation in the L-packet 11^. 

The Vogan bijection between the set 11^ and the group of homomorphisms from 
Ay to (±1) is normalized as follows. Assume that the hermitian space V is split. 



lh = (-i) 
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The character x = 1 of A v gives rise to the non-degenerate hermitian A-lattice L in V, 
with an orthogonal basis whose inner products are units in A. Let Nl be the unipotent 
radical of an Iwahori subgroup of the hyperspecial maximal subgroup K = Aut(L) in 
U(V). The construction of [GGP, §12] over the ring A gives a surjective homomorphism 

/ + fo ■ N L -> A"- 1 + A(-) 

where A(-) is the eigenspace where a = — 1 on A, which consists of the elements of 
trace to A . 

By [DR2], the character x = 1 of corresponds to the unique representation 7Ti 
of in the L-packet of (p which is induced from a generic, cuspidal representation of the 
reductive quotient U n (g) of U(L). All of the generic characters of the unipotent radical 
N(q) of a Borel subgroup of U n (g) are conjugate, and we construct one of them in the 
following manner. 

Let ipa be an additive character of k which is trivial on fc an d the maximal ideal P 
of A, but is nontrivial on A. Since A is unramified over A , we have 

A(+) + A(-) = A + 2 ■ A. 

Hence, for elements z in A{— ), the character 

z ^ i>o{z/2) 

is nontrivial on A(—)/P(—). Let ^ be any additive character that is trivial on P but 
nontrivial on A. Then the composition 

V(E/(n)) • Vo(/oW/2) : JVl S 1 

gives a generic character of A^, whose reduction mod P is a generic character of N(q). 
Hence the representation 7Ti corresponding to the trivial character of A^ is generic 
for the character obtained by scaling the additive character ip used to define the root 
number character in Proposition 13.11 and Corollary l3.2l by the factor 1/2, or equivalently 
by the factor 2. This is the normalization predicted in [GGP, Conjecture 16.3]. 

Now consider the parameter of G — XJ(W) x U(Wq) = U n x U m which is given by 

M = 0M( fti ) 
N = 

From the calculation of the character \ — Xn x Xm of A v in the previous section, we 
conclude that the irreducible representation ir x of G = U(W) x U(Wo) is compactly 
induced from a maximal compact subgroup with reduction isomorphic to 

(Up x U n _ p ) x (Up x U m _p) 



RESTRICTION PROBLEMS FOR CLASSICAL GROUPS 



9 



over F q . Here p > is the number of pairs (aij, Pi) with a.$i = ji. The finite dimensional 
representation that we are inducing has the form 

{R®R{a))®{R w ®R{(3)) 

where R is the Deligne-Lusztig representation of U p (q) associated to the character 
(«!, a 2 , ol p ) and R y is its dual representation, associated to the character /3 2 , P P )- 
The remaining representations R(a) of U n _ p (g) and R(j3) of U m „ p (g) are associated to 
characters whose components and (5j satisfy ctijSj ^ jj, for all i,j. 

As support for [GGP, Conjecture 16.3], we will prove: 

Theorem 3.3. Let n x be the depth zero supercuspidal representation of G = XJ(W) x 
U(W / ) defined above, which corresponds to the distinguished character in [GGP, con- 
jecture 16.3]. Then n x possesses a Bessel model, in the sense that 

dimHonifl-(7r x , v) = 1 

where (H,^) is as defined in [GGP, §12]. 



To prove the existence of a (unique) Bessel model for n x , it is sufficient to establish 
the existence of a Bessel model for the representation 

R(a) <g> R(/3) of U n _ p x U m _ p , 

as there is clearly a unique U p x U p invariant linear form on (R ® R v ). We will do 
this in the following two sections, after first studying the situation for general linear 
groups. 

4. Branching laws for GL n (F q ) 

In this section, we calculate the restriction of a representation of GL„(F g ) to GL„_i(F 9 ) 
where GL„_!(Fg) sits inside GL n (F q ) in the natural way as 

AO 
1 

These branching laws are surely known in the literature, such as in the work of Thoma 
[Th]; however, we have preferred to give a different independent treatment. 

We begin by recalling the notion of twisted Jacquet functor. Let P = M ■ N be 
any group such that iV is a normal subgroup of P and let if be a character of iV 
whose stabilizer in M is denoted by M v . The data (N, </?) defines the twisted Jacquet 
functor from the category of smooth representations of P to the category of smooth 
representations of M v . It associates to a representation V of P the largest quotient 
Vn,<p of V on which iV operates via the character ip; clearly Vn, v is a representation 
space for M v . The twisted Jacquet functor is exact. 
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Now let E n -i be the mirabolic subgroup of GL„(F g ) consisting of matrices whose last 
row is equal to (0, 0, • • • , 0, 1) and let N n be the group of upper triangular unipotent 
matrices in GL n (F g ). We fix a nontrivial character ip of¥ q and let ip n be the character 
of N n , given by 

1p n (u) = ^ («1,2 + «2,3 H h «n-l,n)- 

For a representation n of GL n (F ? ), let 

it 1 = the i-th derivative of tt, 
which is a representation of GL n _i(¥ q ). To recall the definition of 7r*, if 

Rn-i = GL n _j(F g ) ■ Vi 
is the subgroup of GL n (F 9 ) consisting of matrices 




with g G GL„_j(F 9 ), v G M(n — i,i),z G JVj, and if the character ^ of JVj is extended 
to Vj by extending it trivially across M(n — then we have 

If 7r is an irreducible cuspidal representation of GL n (¥ q ), then n l = n for % — 0, and 
7r™ = 1, the trivial representation of the trivial group GL (F 9 ). All the other derivatives 
of 7r are 0. 

The following proposition is from Bernstein- Zelevinsky [BZ], where it was established 
for non-archimedean local fields, but their proof works for finite fields as well. It is 
known as the Leibnitz rule for derivatives. 

Proposition 4.1. For n x a representation of GL ni (F g ) and n 2 of GL„ 2 (F g ) ; we let 
Tii x 7T 2 denote the representation of GL ni+n2 (¥ q ) induced from the corresponding rep- 
resentation of the parabolic subgroup with Levi subgroup GL ni (F q ) x GL n2 (F q ). Then 
there is a composition series of the k-th derivative (tti x 7r 2 ) fc whose successive quotients 
are n{ x n^ 1 for i — 0, • • • ,k. 

Here is a generality from Bernstein and Zelevinsky [BZ]. 

Proposition 4.2. Any representation £ of ' E n -i has a natural filtration of E = E n _i 
modules 

0CE„CE 1 CS 2 C--CS„ 

such that 

S.+x/S, = ind| ® for i = 0, • • • , n - I, 

where Ri = GLj(F g ) • V n ~i is the subgroup o/GL„(Fg) consisting of 
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with g G GLj(F g ), v G M(i, n — i), z G N n -i, an d the character ip n -i on N n _i is extended 
to V n ^i by extending it trivially across M(i, n — i). 

As a consequence of the above two propositions, we have the following corollary. 

Corollary 4.3. Let n = n\ + • • • + n r be a sum of positive integers, and let 7Tj be an 
irreducible cuspidal representation of GL ni (¥ q ) for % = 1, • • • , r. Let 

IT = 7Tl X • ■ • X 7T r 

be the corresponding parabolically induced representation o/GL n (F 9 ). Then the restric- 
tion of ir i x • ■ ■ x 7r r to GL n _i(F,j) is a sum of the following representations: 

TT^ X 7Tj 2 x • • • x n is x E[n - 1 - (z' x H h 

where 1 < z'i < i 2 < ■ ■ ■ < i s < r (the empty sequence is allowed) and i\ H Vi s < n. 

Moreover, 

S[m] = ind^^Vm 

denotes the Gelfand-Graev representation of GL m {¥ q ), with E[l] equal to the regular 
representation ofF* and E[0] denoting the trivial representation of the trivial group. 

Proof. By Proposition 14. 2[ the restriction of II to E n _i is the sum of 

n m /n, = ind^- 1 (n n - l ®Vn-,). 

Since GL n _i(F g ) • Ri = E n _\ for any i, it follows that 

(iWn;)|GL n _ l( F 8 ) = IT 1 -* x E[n - 1 - z], 

where E[n — 1 — z'] is the Gelfand-Graev module of GL n _ 1 _j(F 9 ). It only remains to 
calculate the derivatives IP~* of II, but this follows from Proposition 14.11 □ 

As a simple consequence of this corollary, we have the following. 

Theorem 4.4. Let n = ni + • • • + n r be a sum of positive integers, and let 7Tj be 
an irreducible cuspidal representation of GL n .(F q ), for i = 1, ■ ■ • ,r. Let n — 1 = 
mi + ■ ■ ■ + m s be a sum of positive integers, and let ^ be an irreducible cuspidal 
representation of GL m . (F ? ). Assume that the representations /ii, ■ ■ • , fi s are pairwise 
distinct, so that the corresponding parabolically induced representation [i\ x • • • X \i s of 
GL„_i(F (? ) is irreducible. Then 

dim Hom.GL n _i (""i x • • • x 7r r , \i x x - • • x aO 

is equal to 

s 

JJ(1 + m<) > 1, 

i=i 
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where rrii is the multiplicity with which //, appears in the set {tti • • • , n r }. In particular, 
if the TTi 's are mutually distinct as well, then 

dimHom G L„_i(7Ti x ■ ■ ■ x 7r r , /^i x ■ • ■ x /i s ) = 2 d 
where d is the cardinality of the set 

{7Ti, • • -7T r } H {/il, 

Corollary 4.5. The restriction of the representation tt\ x • • • x 7r r 0/ GL n (F g ) con- 
tains the representation [L\ x • • • x [i s 0/ GL n _i(F 9 ) (with fa's mutually distinct) with 
multiplicity one if and only if the sets {tti, • • • , n r } and {/ii, • • • , fi s } have no common 
elements. 

5. Branching laws for U n (F g ) 

In this section, we use the method of base change, also called Shintani descent, to 
deduce some conclusions about branching laws for the restriction of a representation of 
U n (F g ) to U(n — 1, W q ) from the corresponding results for general linear groups obtained 
in the previous section. The result is then applied to give a proof of Theorem 13.31 

We make crucial use of the multiplicity 1 theorem for restriction of representations of 
unitary groups over p-adic fields, which was recently proved by Aizenbud, Gourevitch, 
Rallis and Schiffmann in [AGRS]. A simple consequence of their result is: 

Proposition 5.1. Let tt\ be an irreducible cuspidal representation o/U n (F g ) and let 

tt 2 = Jp(cr) 

be an irreducible generalized principal series representation of XJ n+ i(W q ) , where P is a 
parabolic subgroup o/U n+ i(F g ) and a is a cuspidal representation of a Levi factor of 
P. We allow the possibility that P = U„ + i, in which case n 2 = a is cuspidal. Then 

dimHom Un(Fg) (7r 2 ,7ri) < 1. 

Proof. Let ko be a local field with ¥ q as its residue field and let k be its unramified 
quadratic extension. Then one can find quasi-split unitary groups U(W ) and U(W) 
with Wo C W, such that U(Wo) x U(W) over ko contains a hyperspecial maximal 
compact subgroup K x K with reductive quotient U n (F g ) x U n+ i(F g ). Moreover, one 
may find a maximal parabolic subgroup P of U(W), such that P R K maps to the 
parabolic P in the reductive quotient U n+ i(F 9 ). 

Let 7Ti be a depth zero supercuspidal representations of U(W / ) which is obtained 
from 7Ti by compact induction, so that 

7Ti = mdj> ; tti = Ind^ 
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Similarly, let a be a depth zero supercuspidal representation of the Levi factor of P 
which contains r as a type. Then we may consider the generalized principal series 
representation Ip{(j) of U(VT) which is irreducible for a generic choice of a. Moreover, 
if K\ denotes the kernel of the natural projection map 

K -» U„ + l(Fg), 

then one has 

Ip(a) K ^=I P (a). 

Now by Frobenius reciprocity, we have 

dimHom u(VFo )(/p(cr),7Ti) = dimHom E - (Jp(5"), 7Ti) 

= dimHom Un(F(j) (/p(o-)^ ' 1 ,7ri) 

where K ^ is the kernel of the projection map K -» U n (F 9 ). Since _K" ,i C Ki, we 
have 

Ip(a) K ^ D Ip(v)* = I P (a). 

Thus we conclude that 

dimHomu(w )(/ P (cr),7h) > Hom Un(Fij )(7r2, tti). 

By [AGRS], the LHS is bounded above by 1 for a generic choice of a (so that Ip((r) is 
irreducible), and hence so is the RHS. This proves the proposition. □ 

A corollary of the above proposition is the uniqueness of Bessel models for cuspidal 
representations of unitary groups over finite fields. 

Proposition 5.2. Let 7Ti be an irreducible cuspidal representation ofU n (¥ q ), and let 
TT2 be an irreducible cuspidal representation ofXJ m (W q ) with n > m but m^n mod 2. 

(i) Let P be a maximal parabolic subgroup of U n+ i(F g ) with Levi factor GL r (F 9 2) x 
U m (F g ) (so that m + 2r = n + 1) and let r be a cuspidal representation of GL r (F ? 2) . 
Consider the generalized principal series representation Ip(r Kl7r 2 ). Then, with the data 
(H, v) defined as in [GGP, $12], we have 

Hom^TTi Ktt 2 ,^) = Hom Un(Fq) (/p(r, 7r 2 ),7r^) 

(ii) We have: 

dimHomp-(7Ti KI 7T2, v) < 1. 

Proof, (i) This is the finite field analog of [GGP, Theorem 15.1], with the same proof. 

(ii) If n = m + 1, (ii) is a special case of Proposition 15.11 In the general case when 
n > m+1, we choose r in the context of (i) so that the induced representation Ip(t^7i 2 ) 
is irreducible. Then (ii) follows immediately from (i) and Proposition 15. 1[ □ 



14 WEE TECK GAN, BENEDICT H. GROSS AND DIPENDRA PRASAD 

After the above propositions, we shall study the restriction problem from XJ n (¥ q ) to 
U„_i(F g ) using Shintani descent. We begin by giving a brief review of this notion. 

Let G be a connected reductive algebraic group over ¥ q and let m > 1 be a fixed 
integer. The group G(¥ qm ) comes equipped with its Frobenius automorphism F, whose 
set of fixed points is G(¥ q ). There is a natural map, called the norm mapping, 

{F-conjugacy classes in G(¥ q m)} — ► {conjugacy classes in G(¥ q )} 

which is a bijection. The norm mapping thus induces an isomorphism of vector spaces 

{class functions on G(¥ q )} — ► {F-class functions on G(¥ qm )}, 

which is called the base change map, and whose inverse is called Shintani descent. 
Furthermore, the base change map is an isometry: 

where Xi an d X2 are class functions on G(¥ g ) which are Shintani descents of the F-class 
functions x'i an d X2 011 G(¥ g m). Here, the inner products are normalized so that 

(!> !)g(f 9 ) = (1, l)c(F qm ) = 1- 

According to Deligne-Lusztig, given a maximal torus T of G defined over ¥ q , and a 
character 

9 : T(¥ q ) -> C x , 

there is a (virtual) representation of G(¥ q ) denoted by R(T,9), which is called a 
Deligne-Lusztig representation. Now given a character 9 as above, one has the character 

d m '■ T m = T(¥ q m) — > C x 

obtained by composing 6 with the norm mapping: T(F qm ) — > T(¥ q ). Thus one may 
consider the Deligne-Lusztig representation R(T rn ,9 rn ). The following lemma is [DL, 
5.16]: 

Lemma 5.3. If G has connected center, then if R(T,9) is irreducible, so is R(T m ,9 m ). 

Henceforth, we assume that G has connected center and R(T, 9) is irreducible. The 
irreducible representation R(T m , 9 m ) is invariant by F and thus can be extended (in two 
ways) to the semi-direct product G(¥ qm ) x (F). For any such extension, the restriction 
of its character to the coset G(¥ q ™) ■ F is a F-class function, and one may consider its 
Shintani descent. The following is a basic fact in the theory of Shintani descent: 

Proposition 5.4. There is an extension of R(T m , 9 m ) whose associated Shintani de- 
scent is the representation R(T,9). 
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Now we can begin our study of the restriction problem for unitary groups over finite 
fields. Let 

*7Ti = R(T,i 
n 2 = R(T>, 9') 

be irreducible Deligne-Lusztig representations of U n (F g ) and U n _i(F g ) respectively, and 
let Xi be the character of 7T». We shall consider the quadratic base change of 7r». By 
Proposition 15.41 there are extensions of the representations 

'R(T 2 ,9 2 ) of GL n (F g2 ); 
R{T^9' 2 ) of GL n _!(F ?2 ), 

whose associated Shintani descents are Xi an d Xi respectively. Fixing such an extension 
in each case, we denote the corresponding character of this distinguished extension by 

Xi* 

It follows that 

2 ' (Xl, X2)GL n _i(F q2 )xi(F) = (Xl-^GLn-iO^a) + (xU^Gk-ifF^)^, 

and thus 

2 • (xU2)gl„_,(f ?2 )x(f) = (Xi,X2)gl„_i(f^) + (Xl,X2)u„_i(F g )- 

Now we observe that: 

(i) the left hand side of this last equality is an even integer; 

(ii) the quantity (x[, X2)GL n _-.(F 2 ) was computed in Theorem 14.41 under the assump- 
tion that R(T 2 , 9' 2 ) is an irreducible representation; 

(hi) the quantity (xi, X2)u n _i(F«2) is equal to or 1 in certain cases, by Proposition 15. II 

Together, these observations allow one to compute (xi, X2)u„_i(f 9 ) in certain situations. 

Namely, let us assume that 7Ti and 7r 2 are irreducible Deligne-Lusztig representations, 
and suppose further that 7r 2 is cuspidal. Then the quadratic base change 7r^ and n' 2 
of 7Ti and 7T2 are are irreducible full principal series representations of GL n (F 9 2) and 
GL n _i(F 9 2). Thus, Theorem 14.41 implies that 



(Xi,X 2 ) 



GL„_!(F o2 ) 



1, if the cuspidal supports of 7r[ and n 2 are disjoint, 
an even integer, otherwise. 



On the other hand, by Proposition 15.11 (xi, X2)u„„i(f 9 ) is either or 1. Therefore we 
get the following theorem as our only option. 
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Theorem 5.5. Let tti and 7r 2 be irreducible Deligne-Lusztig representations and sup- 
pose that 7r 2 is cuspidal. Then 

dimHom Un _ 1 (F 8 )(7ri,7r 2 ) ^ 

if and only if the cuspidal supports of the base change representations tt[ and 7i' 2 are 
disjoint, in which case the Horn space has dimension 1. 

In particular, this theorem completes the proof of Theorem l3.3l Indeed, in the setting 
of Theorem 13.31 we need to show that the distinguished representation tt x = tti x 7r 2 of 
XJ{W) x U(W ) satisfies 

Eom H (n x , v) ^ 0. 

By the argument in the proof of Proposition 15.11 it is sufficient to show that the 
representation 

R(a) ® R(f3) of U„_ p (F g ) x U m „ p (F g ) 

satisfies 

ttom H{¥q) {R(a)®R{l3),v) ^ 0. 
The desired non- vanishing then follows from Proposition 15. 2( i) and the above theorem, 
using the fact that the quadratic base change of R(a) and R((3) have disjoint cuspidal 
support. 

6. LANGLANDS-VOGAN PACKETS FOR SMALL UNITARY GROUPS 

The rest of this paper is devoted to verifying [GGP, Conjecture 16.3] or its variant 
[GGP, Conjecture 20.1] in various low rank examples in the unitary and symplectic 
cases. In this section, we explicate the Langlands-Vogan parameterization of irreducible 
representations of U(V) where V is a hermitian (or skew-hermitian) space over k of 
dimension < 3. 

When dimfc V = 1, the group U(V) is naturally isomorphic to the subgroup k 1 of 
norm one elements in k x , via its scalar action on V. The map 

i h> x/x ff 

gives an isomorphism of k x /k£ with U(V). The only other pure inner form of U(V) is 
the group U(V) where V is obtained from V by scaling the hermitian form on V by 
an element in k£ \ Nk x . 

In this case, an L-parameter for U(V) is a 1-dimensional conjugate-orthogonal rep- 
resentation M of WD{k), which corresponds via local class field theory to a character 
of k x /kQ , and hence to characters xm of U(V) and x'm °f U(V'). The Vogan packet 
associated to M is then 

Ha/ = {xm, x'm}- 

The component group Am is Z/2Z and the trivial character of Am corresponds to the 
character xm of U(V). 
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Now consider the case when dimV = 2. We take V to be the split hermitian space, 
and denote the other rank 2 hermitian space (which is anisotropic) by V . In this case, 
the groups U(V) and U(V) are closely related to the group GL 2 and its inner form 
D x , where D is the unique quaternion division algebra over k . 

More precisely, given a quaternion algebra B over fc (possibly split), we fix an 
embedding 

k ^ B 

of algebras over k and regard B as a 2-dimensional vector space over k via left mul- 
tiplication. All such embeddings of k into B are conjugate under Antk (B) by the 
Skolem-Noether theorem. There is an element b e B (of trace zero) which normal- 
izes k and whose conjugation action on k is the involution a; moreover, all other such 
elements are of the form A • b for A e k. We thus have a decomposition 

B = k ■ 1 + k ■ b. 

Define a nondegenerate hermitian form on B by 

(x, y) = projection of x ■ y onto k • 1, 

where y i— > y is the canonical involution on _B; let Vb be the associated hermitian space. 
If B is split, then Vb is the split hermitian space V, whereas if B is the quaternion 
division algebra D over ko, then Vb is the anisotropic hermitian space V . 

The associated unitary similitude group is given by 

GU(Vb) (B x x A; X )/AA; X 
with an element (b,t) £ B x x k x acting on B by 

(b,t)(x) = txb' 1 . 
The similitude character is given by 

(M) ^ Nf -Mr 1 , 

so that 

TJ(Vb) = {(M) e GU(V B ) : N6 = Nt}. 
Observe that U(Vb) is a subgroup of 

GU + (VB) = ((fi x ) + xA: x )/AA; x , 

where 

(5 X )+ = {b E B x : Nb E Nk x }. 
Indeed, it is easy to see that 

G\] + {V b ) = \]{V b )-Z gv{Vb) , 

where 

^GU(v s) = (£:o X xA; x )/AA; x -A; x 

is the center of GU(Vb). 
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For later purposes, we describe here a nondegenerate rank 1 hermitian subspace of 
V B . Consider the nondegenerate subspace 

L B = k ■ b ^ B 

and observe that its orthogonal complement = k ■ 1 is isomorphic to (1). The 
pointwise stabilizer of in U(B) is the diagonal subgroup 

U(L B ) = k x /k* (B x x k x )/Ak*. 

We now come to the representation theory of \J(V B ). Observing that the L-packets 
of GU(Vb) are all singletons, we take an L-packet of U(V B ) to be the set of irreducible 
constituents of the restriction of an irreducible representation of GU(Vs) to U(Vb). 
Since 

G\] + (V b ) = U{V b )-Z gtj{Vb) , 

we see that the restriction of an irreducible representation of GXJ(V B ) to XJ(V B ) is 
completely determined by its restriction to GXJ + (V B ). In other words, from the repre- 
sentation theoretic point of view, we may work with GXJ + (V B ) in place of XJ(V B ). 

More precisely, if r Kl x is an irreducible representation of 

GXJ(V B ) = (B* xk x )/Ak x , 
then its restriction to GXJ + (V B ) is equal to 

r| (B x)+ K x, 

and it is known that r\mx\+ is either irreducible or is the sum of two inequivalent 
summands. Moreover, the latter holds if and only if r <g> uj k / ko = r, in which case we 
say that r is dihedral with respect to k/k . Then the L-packet of V(V B ) associated to 
r is the set 

^-b,t,x — {( r+ ^ x)|u(y s ) : r+ is an irreducible summand of r|( B x)+}, 
which has cardinality 1 or 2. Observe that if [i is any character of k£ , then 

nB lTl g)( M -lodet),x-(AioN) = nS,T,x- 

If TV is the L-parameter of r, we also write Hb,n,x f° r n_B, T ,x- 

To attach L-parameters to these packets, recall that an L-parameter in this case is 
a two dimensional conjugate-symplectic representation M of WD{k). Now we note: 

Proposition 6.1. (i) Let t Kl x be an irreducible representation of GXJ(V), so that 
u t • xlfcg — 1- If N is the L-parameter of r, then the representation 

M = N\ WD(k) ® X 
ofWD(k) is conjugate-symplectic. 
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(ii) Conversely, any 2- dimensional conjugate- symplectic representation M ofWD(k) 
arises in this way from an irreducible representation tMx of GU(V), which is well- 
defined up to twisting by (p~ l o det) Kl p o N for some character p of k$ . 

Proof, (i) Let 

(-,-): N <g> TV — ► A 2 N = det N 

be the natural skew-symmetric W^-D(/co)-equivariant form. Also, for s G WD(ko) \ 
WD(k), there is a VT-D(A;)-equivariant isomorphism 

N s — >■ A" 

given by the action of on N. By composition, we obtain a W^-D(/c)-equivariant 
bilinear form 

N ® N s N ® N — ► det A/". 
Twisting A" with x and A^ with x s gives a conjugate duality 

B:M®M S — • (let A' • \ • \ " 1. 
where the last equality follows from the fact that 

det N • x \k* = 1 on ^k x . 

To see that this conjugate duality has sign —1, we write pu for the action of WD{k) 
on M and p N for the action of WD(k ) on A" and compute: 

B(n,p M (s 2 )m) 

=x(s 2 ) ■ ■ p N {s) 2 m) 

=x(s 2 ) ■ (n,p N (s)m) 

= ~ X(s 2 ) ■ (p N (s)m,n) 

— ~ x( s2 ) ' det N(s) ■ (m, pAr(s)~ 1 n) 

= - x(s 2 ) • det N(s) ■ B(m, n). 

But if t e k* \ Nk x , then 

x(s 2 )-detN(s)=x\ k *(t)-u T (t) = l. 

This proves (i). 

(ii) Conversely, if M is conjugate-symplectic, then det M is conjugate-orthogonal and 
thus has the form x/x a f° r some character x of WD{k). Moreover, such a x is well- 
determined up to a character of the form pN. The representation M ® x~ X is then 
a-invariant and hence is the restriction to WD(k) of a representation A" of WD (ho). 
Such an A" is not unique, as one can choose to twist any irreducible summand of A" by 
iVk/ko- In any case, we have 

M = N\ WD(k) <8> x, 
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and 

det N • x| fc x =1 or u k/ko . 

We need to show that N can be chosen so that the first possibility holds, and that 
this choice of N is unique up to twisting by u>k/k (for x fixed). For this, we consider 
various cases: 

(a) if iV is reducible, then by twisting one of its irreducible summand (which is 1- 
dimensional) by 0Jk/k if necessary, we can ensure that det N ■ %| fe x = 1. For a fixed 
choice of Xi t ne only other N for which this holds is N ® uJk/k a - 

(b) if M is irreducible, then Schur's lemma implies that any two conjugate dualities 
of M are multiples of each other and thus must have sign —1 in our setting. On 
the other hand, the construction in the proof of (i) gives a conjugate duality on M 
which has sign 

+1, if det N • x| fc * = Uk/k ; 
-1, if detN- X \ k * = I- 
Thus, det N • x| fe x = 1 in this case. 

(c) if M is reducible but N is irreducible, then we have 

M = Mi + M 2 
with Mi and M 2 distinct conjugate-symplectic, and 

M 1 -M 2 = X /X a - 

Moreover, 

JS - \Ti(l WD{k) IVhx 

so that 

detN = x~\. 
In particular, det iV • x\k y = 1 in this well. 
Hence (ii) is proved. □ 

In view of the above proposition, we set the L-parameter associated to the packet 
n_B,r,x to be the conjugate-symplectic representation 

M — N\ W D(k) ® Xi 

with N the L-parameter of r. Given a conjugate-symplectic M, with associated pair 
(t,x) as in Proposition 16. l( ii) . the associated Vogan packet is 

B 

where the union is taken over the two quaternion algebras over ko- 
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Remark: It has been shown by Konno-Konno [KK] that the above construction of 
//-parameters agrees with the one supplied by the theory of twisted endoscopy (i.e. 
base change to GL(2) over k), which has been achieved by Rogawski [Ro] using the 
stable trace formula. 

The following table lists the various possibilities of M, Um and the component group 
Am, depending on the type of r's. 



T 


M 


n M 


A M 


non-dihedral principal series 
(with respect to k/k ) 


P + CT P V , 

p ^ a pV 


1 representation 
on XJ(V) 


trivial 


non-dihedral discrete series 
(with respect to k/k ) 


irreducible 
conjugate-symplectic 


1 representation on U(V) 
and 1 on XJ(V) 


Z/2Z 


dihedral principal series 
(with respect to k/k ) 


2-M', 
M' conjugate-symplectic 


2 representations 
on XJ(V) 


Z/2Z 


dihedral discrete series 
(with respect to k/k ) 


M 1 + M 2 , Mi 7 M 2 
conj ugate-symplectic 


2 representations on U(V) 
and 2 on XJ(V) 


Z/2Z x Z/2Z 



If the conjugate-symplectic representation M is of the last two types in the above table, 
we shall call M dihedral with respect to k/k . If it is of the first two type, we shall call 
it non-dihedral with respect to k/k . 

From the above table, we see that #IIm = = #Rr(^M)- To index the repre- 

sentations in U M by Itt(A m ), we need to fix a generic character of V(V). According 
to [GGP, Prop. 12.1(2)], a generic character of U(V) is specified by giving a nontrivial 
additive character 

ip : k/k -> S 1 . 

Via the description of GXJ(V) in terms of GL 2 (/c ) given above, this then corresponds 
to a generic character of GL 2 (k ), which is given by an additive character 

^° : k -> S 1 . 

To describe the precise relation between ip and ip , we need to start with an explicit 
embedding k — > M 2 (k ). We do this in a standard way, by choosing a trace zero 
element e of k, which gives 

k = k ■ 1 + k ■ e and End k() (k) = M 2 (k ) . 
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The multiplication action of k on itself then gives an embedding k ■=— > M2 (&<))• More- 
over, the action of a on k gives rise to an element b in Endfc (/c), so that 

M 2 {k ) =k + k-b. 

After some calculations with these explicit data, which we will omit here, one sees that 
ip and ip° are related by 

if)(x) = i/; (Tr(e • x)) for all x G k. 

We stress again that the above relation depends crucially on the choice of the trace 
zero element e, though we will not need to make use of this relation in this paper. 

In any case, with ip : k/ko — ► S 1 and hence ip° : ko — > S 1 fixed, we note that 
r| GL2 ( feo )+ has a unique ^-generic constituent and hence the Vogan packet IT M has a 
unique ^-generic element. We then decree that 

(i) the trivial character of Am corresponds to ?/>-generic element in Hm] 

(ii) a character of Am corresponds to a representation of U(V) if and only if it is 
trivial on the image of the central element — 1 e L U(V). 

From the above table, we see that these requirements completely determine the 
bijection 

U M <-> kr(A M ), 

except in the last case, where r is a dihedral (with respect to k/ko) discrete series 
representation of GL 2 (/c ). In that case, if r' denotes the Jacquet-Langlands lift of r 
to D x , then we do not know how to label the two summands of r'\mx\+ using the 
two characters of Am which are nontrivial on the central — 1. However, in §[HJ we shall 
resolve this issue when we describe an alternative construction of these Vogan packets 
using theta correspondence. 

Finally, we consider the case when dim V = 3. In this case, the only other pure inner 
form of U(V) is the group U{V) where V is the hermitian space obtained from V via 
scaling by an element of k£ \ Nk x . In this case, the Vogan packets have been defined 
by Rogawski [Ro] via base change to GL(3) over k using the stable trace formula. 

The L-parameters are conjugate-orthogonal representations M of WD(k) of dimen- 
sion 3. When M is irreducible, the associated Vogan packet is said to be stable; it 
consists of a representation of U(V) and the same representation regarded as a repre- 
sentation of U(V'). The component group A M is Z/2Z and we decree that the trivial 
character corresponds to the representation of U(V). On the other hand, when M is 
reducible, the associated Vogan packet is said to be endoscopic. In §E1 we shall describe 
a construction of the endoscopic packets, and the labelling of their representations by 
Iyy(Am), via the approach of theta correspondence. 
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7. Theta correspondence 

The goal of this section is to review the necessary background and framework for 
the theta correspondence for unitary groups. This is necessary for the construction of 
endoscopic Vogan packets of U(2) and U(3) which will be given in the following section. 

Let V be a hermitian space and W a skew-hermit ian space over k. To consider the 
theta correspondence for the dual pair V(V) x U(W), one requires certain additional 
data: 

(i) an additive character tp : ko — > S 1 ; 

(ii) a character \i : k x — > C x such that fi\ k x = 0Jk/k - 

To elaborate, the tensor product ReSk/k {V ®fc W) has a natural symplectic form 
defined by 

(vi ®w 1 ,v 2 ®w 2 ) = Tr k/ko ({v 1 ,v 2 ) v • (w 1 ,w 2 ) w ). 
Note that many authors (for example [HKS]) include a factor 1/2 on the right hand 
side, but we shall not follow this convention here. In any case, there is a natural map 

i : XJ(V) x XJ(W) — ► Sp(V <g> W/ko). 

One has the metaplectic S 1 -cover Mp(V" <g> W) of Sp(V <8> W), and the character ip 
(together with the form (— , — ) on V <8> W) determines a Weil representation uj^ of 
Mp(y <S> W). To obtain a representation of U{V) x U(W) from o;^ , however, one needs 
to specify a splitting of the map % to the metaplectic cover. This is quite subtle, but 
was completely understood by Gelbart-Rogawski [GRO], Kudla [K] and Harris-Kudla- 
Sweet [HKS]; it requires the additional data above. 

More precisely, the data (V,ijjo,n) determines a splitting 

iv, li ,* --V(W)^M V (V®W), 

whereas the data (W, ipo, A*) determines a splitting 

iw,*i»:U(V)^Mp(y®W) 

whose image commutes with that of iv^^o- ^ n [HKS], the above splittings can be 
defined for the choice of any pair of characters (%, x') of /c x satisfying 

-v/l —i ,dimV j /I _ dimly 

XU X - u k/ko and x U* - u k/ko . 

In their terminology, our splittings are relative to the pair of characters 

x = fi dimV and X ' = fi dimW - 

In particular, by [HKS, Corollary A. 8], a property of this splitting is that the images 
of the centers of U(l / ) and U(W) are identified, so that the theta correspondence we 
consider here preserves central characters. 
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Using the above splittings, one obtains a Weil representation 

of U{V) x U(VF). Moreover, the Weil representation w^ 0)fl depends only on the orbit 
of ip under Nk x . Thus, given an irreducible representation n of U(W), we have its big 
and small theta lift Q^ 0tfJi (n) and 0^ o ^(tt) on V(V). 

It would appear that, by restricting (xi,X2) (as in [HKS]) to have the special form 
taken here, we are losing one degree of freedom. However, this lost degree of freedom 
can be regained by allowing twisting of the theta lifts by 1-dimensional characters of 
U(V), i.e. if we consider 9^ 0tll {jc) <g> (x ° det) as well. 

It is also useful to consider the theta correspondence for similitude groups. Let 

RC GU(V) x GU(W) 

be the subgroup consisting of elements (g, h) such that sim(<7) • sim(/i) = 1. Then the 
Weil representation kty 0)/1 has a natural extension to R. Now observe that 

R C GV + (V) x GV + (W) 

where GXJ + (V) consists of those elements g G GXJ(V) such that sim(g) lies in the 
image of the similitude map of GU(W), and analogously for G\J + (W). Then one may 
consider the induced representation 

"Vo,M — ma R 

of GXJ + (V) x GXJ + (W), which depends only on the orbit of ip under Nk x (and may even 
be independent of i/j in some cases). We can now consider the theta correspondence 
for GU + (l / ) x GU + (W^) associated to ^Vo,m- ■'■ n particular, for a representation tt of 
G\J + (W), we have its big and small theta lifts Q^^(tc) and Q^^ir) on GV + (V). 

In this paper, we will be considering the theta correspondence for U(l / ) x U(W) with 
| dimV — dimIU| < 1. In this case, there are some rather precise conjectures about 
the behavior of the theta correspondence in the literature (see for example [HKS, §7] 
and [P5]). We formulate these as the following working hypothesis. 

Working hypothesis: Let V be a hermitian space and let W be a skew-hermitian 
space, and consider the theta correspondence for U(V) x U(W) relative to the data 
(ipo,/i). For an irreducible representation n of U(l / ), let O^^in) denote the (small) 
theta lift of tt to XJ(W). 

(a) If dimV = dimPU, then the Langlands parameters of tt and ^ 0)/1 (7r) are the same 
(if the latter is nonzero). For a given L-parameter M, the theta correspondence 
induces a permutation of the Vogan packet IIm to itself. This bijection is given by 
translation by a character of the component group A M , as given in [P5]. 
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(b) If dim V = dimly — 1, then the Langlands parameters M of 7r and N of 9^ (h ^(n) 
are related to each other by: 

N = y~ x M + /i dimV \ 
The theta correspondence relative to (ipo, p) gives an injection 

This injection can be naturally described in terms of the characters of the com- 
ponent groups of M and N as follows. Assume for simplicity that p dimV does 
not occur in /z _1 M, so that An = Z/2Z x Am- For an appropriately normalized 
Langlands- Vogan parametrization, the above injection is described by the natural 
map 

Irr(A M ) — > hr(A N ) = {±1} x Itt(A m ) 

given by 

P ^ (e,p) 

where the sign e is completely determined by p and the space W. 

Moreover, as V and W vary over all hermitian and skew-hermit ian spaces of the 
specified dimensions, one has 

V,W 

where the union is disjoint and we ignore the theta lifts which are zero. The 
disjointness of the union is in fact a consequence of the main result of [HKS] on 
theta dichotomy. 

The above working hypothesis can be made more precise, especially its relation with 
the Langlands- Vogan parameterization. In the following, we shall consider the low 
rank cases, with dim V < 2 and dim W < 3. In these cases, we shall verify the above 
working hypothesis in its precise form. We note that these low rank cases are the only 
ones in which the Langlands- Vogan parameterization is fully understood for U(V) and 
U(W). 

For example, statement (a) for dimV = 1 is a result of Moen [Mo], Rogawski 
[Ro2] and Harris-Kudla- Sweet [HKS] (see Theorem 19. II below) . whereas the case when 
dim V = 2 is verified in Theorem 111.21 below. On the other hand, statement (b) for 
dimV = 1 is easy to check, and the case of dimV = 2 is due to Gelbart-Rogawski- 
Soudry [GRS]. 

8. Endoscopic packets and theta correspondence 

The goal of this section is to describe an alternative construction of the endoscopic 
packets of the unitary group U(V), via theta correspondence, when dimV = 2 or 3. 
We shall rely heavily on the framework and notation of the previous two sections. 
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Our first case of interest is the theta correspondence for a skew-hermitian space W 
and a hermitian space V with 

dim W = 1 and dim V = 2. 

We shall use the associated theta correspondence to construct certain Vogan packets 
on U(V). We shall fix once and for all a trace zero element 5 G k x , so that 

6" = -5. 

Recall that in §6j we have given a construction of the rank 2 hermitian spaces Vb in 
terms of quaternion algebras B over k . Suppose that 

M = M 1 + M 2 

is a 2-dimensional conjugate-symplectic representation of WD(k), with Mj conjugate- 
symplectic (but not necessarily distinct). As we explained in the previous section, such 
an M gives rise to a Vogan packet I1m of U(Vb). If we fix an additive character 

ip : k/k Q — ► S 1 

then there should be an associated bijection 

J(V>) : n M <— > Irr(A M ). 

It is the Vogan packet Hm, together with the bijection J(i[)), that we would like to 
construct using theta correspondence. In fact, since the Vogan packets on U(Vb) are 
defined by restriction from GU(Vb), it will be better to consider the theta correspon- 
dence for the similitude groups GU(W / ) x GU + (Vb), with 

GV(W)^k x and GXJ + (V B ) = ((B x ) + x k x )/k*. 

To set up the theta correspondence, we need to fix the data ipo and \i. Since we have 
fixed the trace zero element 5 G k, there is a unique additive character ipo of ko such 
that 

ip(x) = ip (^ ■ Ti(6x)). 
It follows that for any trace zero element x G k, 

ip(x) = ip (6x). 

In the rest of the paper, we shall assume that if) and ipo are related as above, via the 
element 5. 

We set 

W = the rank 1 skew-hermitian space with discriminant S, 
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and let W be the other rank 1 skew-hermitian space. For any a G fc x , we let W a 
denote the rank 1 skew-hermitian space obtained from W by scaling by a. Finally, 
with M = Mi + M 2 as above, we set 

fi = M 1 , 

and let x be any character of k x such that 

X /x a = M 1 ■ M 2 . 

This is possible since Mi • M 2 is a character of k x /k£ . The choice of x is n °t unique 
but any two choices differ by a character of k x which is cr-invariant, or equivalently 
that factors through the norm map to k£ . In any case, we have 

M = V + X/X a ■ 

and the packet Um is obtained by the restriction of tMx, where r is the representation 
of B x with L-parameter 

AT T A WD ( k o) -1 

Now we may consider the theta correspondence associated to the Weil representation 
f^ 0>M of GU(W a ) x GU + (Vb)- Regarding x as a character of GU(W a ), we have the 
theta lift 

®w a ,v B ,ii> Ax) = 8w a ,v B rtoAx) 

on GU + (Vb)- With B x = GL 2 (A:o), the character ip determines a generic character 
of GU + (Vb). We let r + be the constituent of r| GL2 ( fco )+ such that the representation 
r + M x of GU + (Vb) is ^-generic, and let r~ denote the other constituent. We also let 
t' be the Jacquet-Langlands lift of r to D x , if it exists. 

With these notations, we have: 
Proposition 8.1. If B is split, so that V B = V , then 

O^wAx) = t + B x, 
O^wyix) =t~ ^X- 

If B is non-split, so that Vb = V , then 

^i> ,ti,w,v'{x) + Qri>,ii,w',v(x) — T> ^ X) 

where the RHS is interpreted as if r' does not exist. In particular, upon restriction 
to U(V) or\J(V), the set 

{^ ,ix,v,w(x), 9ii>o,n,v,w(x)i 9ii>o,n,v,w(x)i ^ip ,n,v,w(x)} 
is the Vogan packet H M associated to the L-parameter 

M = M 1 + M 2 =v + (i^x/x*- 
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Using the above construction of endoscopic packets of U(V), we can define the bi- 
jection 

J(ip) : n M < ► Irr (A M ), 

as follows. Consider the case when M 1 ^ M 2 , so that A M = Z/2Z x Z/2Z; this is the 
only case where the bijection Um <-> Ih(Am) has some ambiguity. We set 

TT = Oi/, ,H,V,W'(X) 
+ " = ^ A 7',W'(X) 

In other words, the recipe for labelling is that 
where 

1 if B is split; 
— 1, if B is not split, 



< 



ei • e 2 = e(5) 
and 



£2 = ^fc/fcoW- 
Equivalently, if 77 is a a character of Am, then 

if and only if 

r](ai) = e(B) ■ u) k/h) (a), 
77(02) = u k/ko (a). 

We leave it to the reader to verify that under this system of bijections J(ip), the 
various desiderata of the Vogan parameterization listed in [GGP, §9 and §10] are sat- 
isfied. In particular, the trivial character of Am corresponds to the unique -?/>-generic 
representation of the packet, and if ip' belongs to the other N/c x -orbit, then the unique 
■^'-generic representation corresponds to the character 

770(0,) = (-l)M. 

Indeed, when M is irreducible, 770 is trivial, whereas when M = Mi + M2 is reducible, 
then 770 is the character ( ) of Am = Z/2Z x Z/2Z. 

It will be useful to convert the above classification into the setting of rank 2 skew- 
hermitian spaces. Using the trace zero element 5, let Wb,s be the skew-hermitian space 
obtained from V B by scaling by 5, and we shall frequently write Wb for Wb,s- Then 
we have 

GXJ(W B ) = GV(V B ) 
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as subsets of Endfe(S). Moreover, the notions of L-parameters and L-packets are the 
same for \J(V B ) and \J(W B ,s)- The only difference lies in the data needed to specify a 
bijection of a Vogan packet with the set of characters of the component group. In the 
case of Vb, we used an additive character 

ip : k/k — ► S 1 , 

whereas for the case of W B ,s, one needs an additive character of k . However, it is easy 
to check that if a representation n of \J(V B ) is generic with respect to ip, then regarded 
as a representation of \J(W B ,s), vt is generic with respect to the character given by 

^oo(y) = ^(S^y), hryeko, 

or equivalently 

ip(x) = ipoo(5x) = ipo (-Tr(5x)) for trace zero x £ k. 

In other words, the character ^oo is precisely the character ip which we have fixed, 
and the bijection 

J($) : n M < ► Itt(A m ) 

for U(Vb) is the bijection J(ipo) for XJ(Wb,s)- For a character rj of A M , we then have 

TTjj = 9ipo,/J.,W B , s ,Va(x) 

where fi and x are obtained from M as before, V a is the rank 1 hermitian space with 
discriminant a, and 

77(01) = e(B) ■ uj k /k (a) 
rj(a 2 ) = ^fc/ fco (a). 

Finally, we consider the endoscopic Vogan packets of U(V) when dim V — 3. Hence, 
we fix a rank 3 hermitian space V and let V denote the other rank 3 hermitian space. 
More generally, for any a G k£ , we let V a denote the hermitian space obtained from V 
by scaling the hermitian form by a. 

Consider L-parameters of U(V) of the form 

M M. ■ M 2 

where Mj are conjugate-orthogonal representations of WD(k) of dimension i. Unless 
M = 3- Mi, we may further assume that Mi is distinct from any irreducible constituent 
of M 2 . It was shown in [GRS] that the Vogan packet n M can by constructed using theta 
correspondence from XJ(W B ), where Wb is the rank 2 skew-hermitian space introduced 
above, together with twisting by 1-dimensional characters of U(V). 

To specify the data needed for theta correspondence, note that M x is a character 
of k x /k£ and thus we can hope to find a conjugate-symplectic character /i of k x such 
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that 

Mi = /i 2 . 

This cannot always be achieved, but if we allow ourselves to replace M by a twist, 
then it can certainly be done. Indeed, one would simply pick a conjugate-symplectic 
fi and twist M by /^M-f 1 . Since the Langlands-Vogan parametrization is compatible 
with twisting, there is no loss of generality in assuming that Mi has a square root ji 
which is conjugate-symplectic. 

Now set 

N = M 2 - n 

so that N is conjugate-symplectic and is an L-parameter for XJ(W B ), and 

M = /i 2 + iV • (T 1 . 

For the additive character ip of k Q , we can now consider the theta correspondence 
associated to the Weil representation ^ ^w B ,v a - 

More precisely, let Hat be the Vogan packet associated to N, together with the 
bijection 

J(V> ) : < ► Irr(A Ar ) 

associated to the additive character ip . Then for r\ e Irr(Ajv), we may consider the 
theta lift 

where n v G IT^r is the representation of XJ(Wb) (this uniquely specifies B) indexed by 
rj under J(tpo). As the element a varies over the two representatives of k£ /Nk x , and 
the character t] varies over Itt(A^), we obtain a collection of 2 • #II/v representations 
(possibly zero). It was shown by Gelbart-Rogawski-Soudry [GRS] that this set of 
representations so obtained is the Vogan packet associated to the endoscopic parameter 

n M . 

The following lemma, which was shown in [GRS], addresses more precisely the issue 
of non-vanishing of these theta lifts. 

Lemma 8.2. Let M = Mi + M 2 = + N ■ /z -1 as above. If M £ 3M U assume without 
loss of generality that Mi is distinct from any irreducible constituent of M 2 . 

(i) If M ^ 3Mi, then the representation ^ 0lM ,w s ,v a ( 7r »;) is always nonzero. 

(ii) If M = 3Mi, then N = 2 • /i 3 and = Z/2Z, so that we may regard r\ = ±1, 
depending on whether i] is trivial or not. The representation 0^ o ^w B ,v a {^ri) is nonzero 
if and only if 

uj k/ko (discV a ) = rj. 
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In each case above, the non-zero representations are mutually distinct. Moreover, the 
representation ^Vo^Wb,^ 71 ^) is generic if and only if n v is generic with respect to 

■0O,disc(V a ) • 



W = 



We may now define a labeling of the elements in Um by the irreducible characters 
of A M . 

(i) If M ¥ 3Mi, and M 1 does not occur in M 2 , then 

A M = A Ml x A M , 2 = A Ml x A N . 
For a character x — ( € :V) e irr (^Mi) x Irr(Ajv), we set 

with 

e ■ rj(-l) = u k/ko (a), 

and 

V, if u k/ko (discV) = 1; 
77 • 77^,0, if w fc /feo(discy) = -1, 

where 77^,0 is the character of An which indexes the -^'-generic element of IIjv- 
More simply when disc(V) = 1, we have 

X(ai) = uj k/ko (a) ■ rj(-l) = uj k/ko (a) ■ e{B) 

and 

x\a M2 = V- 

In particular, for a character x of A M = A Ml x A N , ir x is a representation of U(V) 
if and only if x(~ 1, —1) = 1. 

(ii) If M = 3Mi = 3/i 2 , then 

A M = A N = Z/2Z. 
For a character 77 = ± of A M) we set 

7^ = 0fo^,W B ,V a (^vu> k/ko (discV)) 

with 

^fe/fc ( a ) = *7- 

By part (ii) of the above lemma, this condition ensures that the theta lift 
above is nonzero. In particular, the trivial character of Am corresponds to a 
representation of U(V) whereas the nontrivial character corresponds to the same 
representation regarded on U(V). 
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Note that since dim!/ = 3, there is only one orbit of generic characters for U(V), 
and hence the Vogan parameterization in this case is canonical. So it is instructive 
to observe that the above parameterization is independent of the choice of ipo. We 
leave this to the reader, as well as the verification that the above definition satisfies 
the desiderata of the Vogan parameterization listed in [GGP, §9 and §10]. 

9. Skew-hermitian case: U(l) x U(l) 

Having explicated the Langlands- Vogan parameterization of the unitary groups U(V) 
with dim V < 3, we are now in a position to verify instances of [GGP, Conjecture 16.3]. 

In this section, we consider the case of a pair of skew-hermitian spaces W C V, with 
dimW^ = dimV = 1. Without loss of generality, we assume that 

5 = the discriminant of V. 

As in the previous section, we let W = V be the other rank 1 skew-hermitian space, 
and more generally, for any a G k£ , we let W a be the skew-hermitian space obtained 
from W by scaling by a. 

To specify the restriction problem in this setting, fix an additive character 

: k -> S 1 , 

and a character \x of k x with 

These determine a Weil representation ujw a ,^ ,^ of XJ(W a ). Also, fix two conjugate- 
orthogonal representations M and N of dimension 1, which gives rise to characters 
a x (3 of U(W ) x U(W ). We are interested in determining 

Hom u(Wa) (a • f3, io Wa ^ (hlI ). 

This question has been resolved by Moen [Mo], Rogawski [Ro2] and Harris-Kudla- Sweet 
[HKS], and we state the result from [HKS, Corollary 8.5] as: 

Theorem 9.1. For each a G k£ , let W a be the rank 1 hermitian space with discriminant 
a ■ 5, and for each b G , let be the rank 1 hermitian space with discriminant b. 
Given a character r] of k x /k$ , which can be regarded as a character of\J(W a ), we have 

Rom v{Wa) (r),u Wa y b ^) ^ e(r? • /i" 1 , ip (Tr(5-))) = u k/ko (a ■ b). 

Remark: We note that our convention here differs from [HKS] in two aspects. Namely, 
we have adopted the convention that on W a <S> H> the symplectic form is Tr((— , —)w a ® 
(— , — )yj. In [HKS], the symplectic form is 

^•Tr(<-,->^®<- -) Vb ). 

Besides the factor of 1/2, the skew-hermitian form on W a is conjugated by a, which is 
necessitated by the convention adopted by [HKS] that skew-hermitian forms are linear 
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in the second variable and hermitian forms are linear in the first variable. Conjugating 
the form on W a by a has the effect of replacing 5 by —5 in [HKS, Corollary 8.5]. 

To apply the above theorem to [GGP, Conjecture 16.3], set r] = a ■ (3 in the theorem, 
and note that the distinguished character xo of Am x ^-n — Z/2Z x Z/2Z given in 
[GGP, Conjecture 16.3] satisfies 

Xo (-l, 1) = Xo(l, -1) = e(M ® A^T 1 ), ^ (Tr(5-)). 

Thus, Theorem 19.11 implies that 

Xo is trivial Rom v{w) (a ■ (3, uj w>i , 0>fl ) ^ 

and 

Xo is nontrivial -<=^ Hom U ( H /)(a / • f3\u W '^ ^) ^ 0. 
This verifies [GGP, Conjecture 16.3] for this case. 

10. Restriction from U(2) to U(l) 

In this section, we consider the restriction problem from U(2) to U(l). This problem 
has been studied by H. Saito [Sa2] and T. Konno [Ko], but we shall give an independent 
treatment here and relate the result to [GGP, Conjecture 16.3]. 

Recall that in £j6l we have given a construction of rank 2 hermitian spaces V# using 
quaternion algebras B over ko, together with a non-degenerate rank 1 subspace: 

L B ^ V B , 

such that 

Li = (i). 

When B is split, this gives a pair of split hermitian spaces L C V, with 

disc(L) = -1. 

On the other hand, if B is the quaternion division algebra D, one obtains a relevant 
pair L' C V with V anisotropic. The groups 

G = G(V) x G{L) and G' = G{V) x G(L') 

are relevant pure inner forms of each other. 

Suppose that M is a conjugate-symplectic 2-dimensional representation of WD{k), 
with component group A M , so that M determines a Vogan packet I1 M of U(V). We 
fix an additive character ip of k/k , and translate it by —2 • disc(L) = 2, using the 
resulting character ip 2 to fix the bijection 

J(ip 2 ) ■ n M «-> Itt(Am). 
Recall that the parameter M gives rise to a representation 

t H x of GU(y) = (GL 2 (A; ) x k x )/Ak^. 
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If t' denotes the Jacquet-Langlands lift of r to D x , then Um is simply the set of 
irreducible constituents in the restriction of r M x and r' M x to U(V) and U(V) 
respectively. 

Similarly, suppose that N is a 1-dimensional conjugate-orthogonal representation of 
WD(k). Then N determines a character rj of k x /k^, which may be regarded as a 
character of U(Lb). We are interested in determining 

Hom u(Li3) (7r B <g>7?,C) 

for n B G Hm,b and rj the character of U(L B ) corresponding to N. 

Since the embedding 

U(L B )^XJ(V B ) C G£/ + (Vb) 
is given by the diagonal map 

k x /k* (£ x x A; X )/AA; X , 

we see that 

Hom u(Ls) ( 7 r i? ®r ? ,C) 
-B 7r S 6n MiB 

= Hom fc x (r, x~ V 1 ) + Hom fc x (r', X~ V 1 )- 

Now we note the following theorem of Waldspurger [Wa2], Tunnell [Tu] and Saito [Sa]: 

Theorem 10.1. Let r be a representation of GL 2 (A:o) with L -parameter N(r) and 
Jacquet-Langlands lift r' on D x . For any character v of k x , with u\ k x = uj t , we have 

dim Hom^x (r, v) + dim Hom^x [j\v) = 1. 

Moreover, 

Hom feX (r, v) ^ e{N{r)\wD{k) ® v~ l = 1. 

Applying this theorem to the case at hand, with v = x 1 ' V 1 ^ we immediately 
deduce [GGP, Conjecture 16.1] (multiplicity one in L-packets). In fact, when r is 
not dihedral with respect to k/k , this theorem also implies [GGP, Conjecture 16.3]. 
Indeed, in this case, r M x remains irreducible when restricted to U(V), so that 

11m = {ttm, 7Tm}- 

Moreover, A M = A N = Z/2Z and the distinguished character xo of A M x A N satisfies 

Xo(-M) = Xo(l,-l) = e{N(T)\ WD{k) ® X -r]^)- 
Hence we deduce that 

Xo is trivial -<=>- Homu(L)(7rM <8>r},C) ^ 

and 

Xo is nontrivial •<=>■ Hom U (L')( 7r M ® C) 7^ 0. 
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Suppose then that r is dihedral with respect to k/ko, so that 

N{T)\ WD(k) = a + a a 

for a character a of P. In this case, r is the sum of two distinct irreducible summands 
when restricted to GL 2 (/kj) + and the same holds for its Jacquet-Langlands lift r' (if 
it exists). Now we have the following refinement of Theorem 110.11 due to the third 
author [P3]. 

Theorem 10.2. Fix an additive character^ of k/ko- Suppose that r is dihedral with 
respect to k/ko. 

(i) The two irreducible summands o/TGL 2 (fc ) + can be indexed as r ++ and r such that 
the following holds. For any character (3 of k x with 

P\k* - w r = a|fc x " ^fc/fco, 

we have 

e(a'./T\V) = +l, 
e(a a ■ (3-\iIj) = -1. 

(ii) Similarly, the two irreducible summands of r'\mx)+ can be labelled as (V) + ~ and 
(t')~ + such that the following holds. For any character (3 of k x as above, 

Hom fc x((r') + ~,/3) ^ e(a -Z?- 1 ,^) = +1 and e(a a ■ /T 1 , ip) = -1, 

and 

Hom fc x((r')" + ,/3) ^ e(a -/T 1 ,^) = -1 and e{a a ■ P' 1 , tp) = +1. 

This theorem immediately implies [GGP, Conjecture 20.1], which is a variant of 
[GGP, Conjecture 16.3]. To obtain the full [GGP, Conjecture 16.3], one would need to 
relate the labelling supplied by the above theorem with the bijection I1 M <-> Itt(Am) 
determined by the additive character ip 2 - Recall that this bijection was constructed in 
$8] using theta correspondence. 

To recall the construction, note that when r is dihedral with respect to k/ko, we 
have 

M = Mi + M 2 

with Mi conjugate-symplectic (not necessarily distinct). We assume that Mi ^ M 2 , 
since the case Mi = M 2 is similar. Then we have 

A M = Z/2Zai x Z/2Za 2 . 

Setting 

fi = Mi and xlx° = Mi ■ M 2 , 



and 



Rom k , (r ++ , f3) + <^ e{a ■ /T 1 , ip) 
Hom fcx (r~, /?) ^ e(a • /T 1 , V) 
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the packet Um consists of the representations 

n P = 0if>o,2,n,v B ,w a (x) 

where W is the rank 1 skew-hermitian space of discriminant 5, and ip and tpo are related 
by 

Moreover, we have 

p(ai) = e{B) ■ uj k/k0 (a) 

and 

p(a 2 ) = uj k/ko (a). 
Now we consider the see-saw diagram 

U(L_ 6 + Li) U(W )xU(W a ) 



U(L_ 6 ) x U(L0 AU(W a ) 

where L_6 denotes the rank 1 hermitian space with discriminant —b. Note that the 
rank 2 hermitian space L_5 + L\ is isomorphic to Vb with 

e(B) = u k/ko (b), 

and the pair L_5 C L_6 + L\ is isomorphic to Lb C Vb We start with the represen- 
tation x|u(w a ) on AU(W a ) and the character on U(L_&), and consider the theta 
correspondence with respect to the additive character ^0,2 ■ Then the seesaw identity 
gives 

Hom u(L _ 6) (7r p ,^ 1 ) = Homu( Wo )(^-i )MiWo)£ _ i (77" 1 ) ® w^ i2lM ,w ,x)- 

Hence, 

Hom u(L _ b) (7r p ,?7" 1 ) ^ 
if and only if the following two conditions hold: 

(a) 

in which case, ^^w^l^^ -1 ) = ^T 1 ; 

(b) 
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But both (a) and (b) are special cases of Theorem 19. II [HKS. Corollary 8.5]. We deduce 
that (a) holds if and only if 

e(/^ -: V\ ^o,2(Tr(5-)) = u k/ko (-b) ■ uj k/ko (a) 

or equivalently 

e(Mi ® N, ijj) = u k/ko (b) ■ u k/ko (a) = p{a x ). 
Similarly, (b) holds if and only if 

eO _1 •^•x/x ff ,V ; o,2(Tr(5-))) = u k/ko (a), 

or equivalently 

e(M 2 ®N,if}) = u k / ko (a) = p(a 2 ). 

Thus, we conclude that 

Homu^)^,^" 1 ) ^ 
if and only if p is the distinguished character xo of [GGP, Conjecture 16.3], where the 
local root numbers are computed using the additive character if) of k/k^. 

11. THETA CORRESPONDENCE FOR U(2) X U(2) 

Before moving on to the next case of [GGP, Conjecture 16.3], we need to establish 
some results about the theta correspondence for U(2) x U(2). More precisely, let Vb be 
the rank 2 hermitian space introduced in £|6j and let Wb 1 be the rank 2 skew-hermit ian 
space obtained from Vb> be scaling by the trace zero element 5 G k x . Then we are 
interested in establishing the theta correspondence for the dual pair 

U(Vb) x U(Wb') 

relative to the data (ipo,p). 

The first result is the following proposition due to Harris [Ha, Lemma 4.3.3] and 
Konno-Konno [KK, Prop. 5.3 and Thm. 5.4]. 

Proposition 11.1. Let M be a 2- dimensional conjugate-symplectic representation of 
WD(k) which gives rise to a L-packet Hm,b for U(Vb) and Hm,b> for XJ(Wb')- 

(i) For any n G U M>B; 

Note that the root number above is independent of the choice of the additive character 
ip ofk/k . 

(ii) If the condition of (i) holds, then 9^ t v B> w B i,n{^) belongs to Hm,b>- In other words, 
the theta correspondence is the identity map on L-parameters. 

Thus, under the theta correspondence for (ipQ,p,), there is a unique B' such that the 
theta lift gives a bijection 

Rm,b < — ► n MjS /. 
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If the parameter M is non-dihedral (with respect to k/ko), then #ILm,b = or 1. 
Hence the above proposition completely determines the theta lift of the representations 
in H M . When M is dihedral with respect to k/ko, then #II MiB = or 2, and in the 
latter case, there are two possible bijections 

which the above proposition does not resolve. In [P5], the third author has given a 
precise conjecture addressing this issue. The following theorem confirms the conjecture 
in [P5] for this case: 

Theorem 11.2. Suppose that M = Mi + M 2 is dihedral with respect to k/ko. Fix the 
additive character tp of k/ko which gives bijections 

fI M < > Irr(A M ), 

and let ip be the additive character of ko such that 

rj){x) = i/) (- • Tr(Sx)). 

Then the permutation o/IIm induced by the theta correspondence associated to (ipo, At, 5) 
is given by multiplication by the character p of A M defined by 

Po(<0 = e(Mi (g)/i~ 2 ,^ 2 ) 

with 

ij>2(x) = ipo(Tr(5x)). 

Proof. Consider first the case where B' is split whereas B is arbitrary. In this case, 
then two elements in IIm,b' can be distinguished by the Whittaker models they support. 
Computing Whittaker models of the Weil representation <^ip ,v B ,w B ,,n, on e sees that for 
?Tp G IIm,b, the representation 9^ y Bt w B ,,nijp) of XJ(Wb') is V'o-generic if and only if 

Hom u(Ls) (7r^,^ 2 ) ^ 0. 

By the result of the previous section, this holds if and only if 

p(a l ) = e(Mi (S>/i _2 ,^ 2 ) and p(a 2 ) = e(M 2 ® fi~ 2 , tp 2 ), 

as desired. This establishes the result when one of B or B' is split. 

The only remaining case is where B and B' are both non-split, so that 

e(Mi ® p~ 2 , ■ e(M 2 ® p~ 2 , if>) = 1. 

In this case, the desired result can be proved by a global method. We give a brief 
sketch of this. 

Let n be a representation in Hm,b, so that 0^ o ^(ti) belongs to IIm,b also. We may 
find: 
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(1) a number field F with at least one real place and such that F Vo = k for 
some finite place v of F; 

(2) an additive character \l> of A F /F such that fy VQ = t/j ; 

(3) a quadratic extension E of F such that E Voo = C and E Vo = k; 

(4) a trace zero element A e E such that A„ = 5; 

(5) an idele class character T, of E x such that S^ = /i and S| A x — uje/f] 

(6) a global quaternion algebra B over F ramified precisely at {foo,^o}, so that 
B„ = B; this gives a hermitian space Vr over F which is isomorphic to Vb over 
F ■ 

(7) a cuspidal representation II of U(Vb) such that 

(a) n„ = tt; 

(b) II belongs to a global endoscopic packet; 

(c) L{BC E/F {H) ® S- 2 , 1/2) ^ 

In particular, it follows that the set 

S = {v : 6(5^,(11,,) ® E v , V) = -1} 

has even cardinality and does not contain the place t>o. Let B' be the quaternion algebra 
over F such that 

e(B;) ^ e(M v ) ■<=>■ v E S. 

In other words, B' is obtained from B by switching the local invariants of B at the set 
S. Since v ^ S, we have 

K = B. 

Moreover, for each place v of F, 

By [Ha], The non- vanishing of the central L- value above implies that the global theta 
lift is non-vanishing as well: 

Now the assertion of the theorem has been checked for all finite places of F outside 
i>o, since at least one of B„ or B^ is split at any v ^ v . At the archimedean places, the 
result of the theorem is also known (c.f. [Pa] for example). If the result of the theorem 
is not true at the place vo, we would have a cuspidal representation 6*,s,y B ,w B ,(n) of 
U(Wb') which violates the Langlands- Arthur multiplicity formula for global endoscopic 
packets of U(2). This gives the desired contradiction. 

For example, suppose that S is empty so that B = B'. Then if the result of the 
theorem holds at all but fails at v , the cuspidal representation Q^ t T,,v m ,w w (Jf) 
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of U(W / b') would differ from the cuspidal representation II at an odd number of places 
v. This is a contradiction. 

□ 



12. Trilinear forms for U(2) 

In this section, we return to the skew-hermit ian case of [GGP, Conjecture 16.3]. In 
particular, we consider the case when 

W CV with dim W = dim V = 2. 

Thus, let Wb,s be the rank 2 skew-hermitian case obtained from Vb by scaling by 5. 
Fix an additive character tpo of ko, and a character fi of k x so that 

This determines Weil representations ctty 0)/x for U(Wb). Given two conjugate-symplectic 
representations M and N of dimension 2, with corresponding Vogan packet Um and 
IIjv, we are interested in computing 

Hom U(V y s )(7r A f ® 7TAT ® Uty^, C) 

as 7Tj\/ and 7Tat vary over all representations in 11m and Hat. 

Note that the representation cj^ 0iAt is not an irreducible representation of U(Wb)- 
However, we may decompose c^ 0)A , according to central characters 

x 

as x runs over characters of Zxs(w B ) = k x /k£ . In fact, this decomposition is simply the 
decomposition of the Weil representation for the dual pair U(Vi) x XJ(Wb)- Thus, each 
summand uty 0iAt [x] is an irreducible representation of XJ(Wb)- Moreover, it belongs to 
an endoscopic packet of XJ(Wb) constructed in Proposition 18.11 

Now, because of central character reasons, it is clear that 



Hom u(Wfl )(vr M ® ix N <g> aty, )M [x], C) = 

unless 

det M ■ det N = x- 

For this \i we have 



Homu(W B )(7TM ® VTat (g) ^ QiAt , C) = iTom u(W / s) (7T M <g> 7TJV <8> C)- 

In particular, [GGP, Conjecture 16.3] amounts to a question about invariant trilinear 
forms on XJ(Wb)- 
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Given that the group XJ(Wb) can be described in terms of GL 2 (/c ) and its inner form, 
we shall see this question can be related to a question about invariant trilinear forms 
for GL 2 . This has been addressed in a series of paper by the third author [PI, 2, 6]; we 
recall his result here: 

Theorem 12.1. Let Ni, N 2 and N 3 be 2- dimensional representations ofWD(k ), with 
associated representations 7ii t B of B x . Assume that det N\ ■ det N 2 ■ det N 3 — 1. Then 



Hom B x (n liB 7i 2: b ® n 3jB , C) ^ e(Aq <g> N 2 <g> JV 3 , ip) = e(B). 

To apply this theorem to the case of \J(Wb), we need to consider the group (B x ) 
and calculate 



More generally, let G be a subgroup of GL 2 (A;o) containing SL 2 (k ). The group G is 
uniquely determined by the subgroup 



consisting of determinants of elements of G. Thus, for any quaternion algebra B, it 
makes sense to define a corresponding subgroup G B inside B x containing SLi(-B). Re- 
stricting representations of B x to Gb, one gets a notion of L-packet of representations 
of Gb- It is known that representations of GL 2 (/co) restrict to G with multiplicity 1, 
but this need not be the case for representations of B x if i? is non-split. For a rep- 
resentation n B of Gb, let m("7TB) denote the multiplicity with which it appears in the 
restriction of an irreducible representation of B x . 

Now we have: 

Theorem 12.2. For i = 1, 2 and 3, let iVj be a 2-dimensional representation of 
WD(ko) with associated representation 7tb,i of B x . Assume that FJ^detiVj = 1. Then 




Moreover, 



dimHom( B x)+(7ri <g> 7r 2 <g> 7r 3 , C). 




^dimHom Gs (7f Sjl (g)7f B)2 ®7f B> 3,C) = #(k x /k£ 2 k x ). 



B 



In particular, 




m(nB,i) ■ m(ir B ,2) • m{n B ,z) • dimHom GB (7r B ,i <S> Kb,2 <S> 7r B ,3,C) 



is equal to 
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where the inner sum is taken over irreducible representations ttb,? of Gb which are 
contained in the representations 7Tj of B x . 

Proof. Clearly, 

Hom Gs (7T Bi i ® 7T Bj2 ® 7f Si3 , C) = HoHl GB (7r Bj i <g> 7T B)2 <g> 7T B>3 , C % ), 

where the x' s range over characters of 5 X trivial on Gb, and C x denotes the 1- 
dimensional representation \ o N B of _B X . But by Theorem 112.14 we have 



y^dimHom Gfl (7f B| i <g> 7r B , 2 ® ttb,3, C x ) = 1, 



for all characters x of order < 2 (by absorbing x in one of the 7iVs). Adding up the 
contribution of the various x's ; we get the conclusion of the theorem. 

□ 

Specializing this theorem to the case Gb = (B x ) + and noting that, in this case, 
m (^B,i) — 1 f° r each B, we obtain: 

Corollary 12.3. In the context of Theorem \12.2\ let G = GL 2 (A;o) + . Then one has, 

dim Hom Gs (ir B ,i <8> 7r S)2 <8> 715,3, C) = 2, 

B Ts,li' r S,2j' r S,3 

where the inner sum is taken over irreducible representations ir B ,i of Gb which are 
contained in the representations txb/i °f B x . 

We can now apply the corollary to the group GXJ + (Wb) or equivalently XJ(Wb)- 

Corollary 12.4. Let Mi be conjugate- symplectic representations ofWD(k) with asso- 
ciated L-packet Hm^b of XJ(W B )- Assume that det Mi ■ det M 2 ■ det M 3 = 1. Then 

ft) 

2j /J dimHom U ( W / B )(7Ti (g> 7r 2 <g> vr 3 , C) = 2. 

B iri6n Mi ,B 

(ii) If one of the Mi 's, say Mi, is dihedral with respect to k/k , so that #Hm u b = 2 
for Bq split, then 

dimHom U ( H / B )(7ri <g> 7r 2 (g) 7r 3 , C) < 1 
for each B. If the above Horn space is nonzero, then 

dimHom U ( H / s ,)(7rJ <g> tt' 2 (g> 7T 3 , C) = 

/or B' ^ 5. 
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Proof. The first assertion follows immediately from the previous corollary and the 
definition of L-packets for U(W) given in §HJ To deduce the last assertion, note that if 

Hom u(l4 / s )(7ri ® 7T 2 (g> 7r 3 , C) ^ 0, 

then we also have 

Hom u(Ws) « <g> tt| <g> n c 3 , C) ^ 0, 
where tt? denotes the conjugate of 7Tj by an element c G GU(Ws) n GU + (W / b)- Since 

dimHom U ( W / B )(7Ti <8> ^2 <8> 7r 3 , C) + dimHoniu^)^ <8> ^ <8> 7r|, C) < 2, 
each of these dimensions must be equal to 1, and all other Horn spaces must be 0. □ 

Remark: Since k^/k^ 2 is a 2-group whose cardinality can be made arbitrarily large 
by choosing k appropriately, and since the L-packet of representations of SL 2 (A;) is 
bounded by 4 [LL], it follows that 

dimHom S L 2 (fc)(>i <g> n 2 <g> 7r 3 , C) 

can be made arbitrarily large. 

Now we can return to [GGP, Conjecture 16.3], so that M and N are two 2-dimensional 
conjugate-symplectic representations of WD{k) which determine Vogan packets Um 
and IT jv of U(Wb)- For a fixed additive character ip of ko, we have obtained a bijec- 
tion 

J(V> ) = n M < > lrr{A M ) 

and similarly for II jy- We are interested in computing 

Hom u(V y s) (vr A/ <g> vr^ <g> ux^) 

for ttm G n M and 7Tjv G Iljv. 

If M and iV are non-dihedral (with respect to kj ko), so that Um and ITv both contain 
at most one representation of each XJ(Wb) (as B varies), then [GGP, Conjecture 16.3] 
is a consequence of Theorem 112.11 Indeed, we have 

A M x A N = Z/2Z x Z/2Z 

and the distinguished character xo satisfies 

Xo(-l, 1) = Xo(l, -1) = e(M ® iV^- 1 ), V) 

for any character ip of k/ko- On the other hand, if Um is obtained by the restriction of 
the representation t m M xm of GXJ(Wb) and LTv is obtained from t n M xn, then the 
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epsilon factor occurring in Theorem 112.11 is 

e(pM ® Pn ® Ind(/x _1 XMXAf), V>o) 

=e(M®N(ji- 1 ),MT±)) 

This verifies [GGP, Conjecture 16.3] in this case. 

When at least one of M or N is dihedral with respect to k/ko, we may appeal to 
the theta correspondence. Since the case when exactly one of them is dihedral with 
respect to k/ko is similar and easier, we shall give the details only when both M and 
N are dihedral with respect to k/ko Thus, let 

M = Mi + M 2 and N = Ni + N 2 , 

with Mi and Ni conjugate-symplectic (not necessarily distinct), and write their com- 
ponent groups as 

A M = Z/2Zei x Z/2Ze 2 and A N = Z/2Z/i x Z/2Z/ 2 . 

In this case, the packet Um can be obtained by theta correspondence from U(l). Set 

v = M X 

and 

Mi • M 2 = -q/rf, 

for some character r] of k x . If L a denote the rank 1 hermitian space with discriminant 
a, then 

n M = {0^,u,w Bit ,L a (v\v(L a )) ■ a e k^/Nk x , e(B) = ±1}. 
Relative to the additive character if> of k/ko, we have the labelling 

71 PM = 9fo,v,W B>s ,L a (T]\TJ(,L a )) 

if and only if 

PAf(ei) = e(B) ■ uJ k / ko (a) and p A /(e 2 ) = u k / ko (a). 
Similarly, a representation in II jv has the form tc Pn , so that 

PN(fi)'PN(h) = e(B). 
Now consider the see-saw diagram 
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U(L a + L_ x ) XJ(W B ) x U(Wb) 



45 



U(L a ) x U(L_0 AU(W S ) 

and note that the rank 2 hermitian space L a + L_i is isomorphic to Vb' with e(-B') = 
w fc/fe ( a )- We start with the representation 77|u(L a ) °f U(L a ), so that the representation 
we obtain on \](W B ) is precisely 

On the other side of the see-saw, we start with the representation // • v • tx" 1 of U(Wg). 
Note that taking contragredient has the following effect on the Vogan parameterization: 
for any character pN of An, the representation tt Pn has Vogan parameter 

(N\p N -(3 ) 

where (3q is the character of A^v = An given by 

Po(bi) = u k/ko (-l). 



Now the see-saw identity gives: 



Hom u(ia) (e^ 0j! ,2 iW/S)ia+L _ 1 (/iz/7rJ jv ),r7| u(La) ). 



Since 



\ 1 ,^ B = ^-^-i,^ » ^ = ^o,P <» 

we see that the LHS of this identity is equal to the desired space 

Hom u(H / B )(7r PM <g> tt Pn <g> uJ^, C). 

On the other hand, the RHS is nonzero if and only if conditions (a) and (b) below are 
satisfied: 

(a) O^o^Ws.La+L-i {.^{.^pnY) 7^ 0- According to Theorem lll.2l this holds if and only 
if 

e(iV v <g> puu~ 2 , if>) = e{B) ■ u k/ko {a), 

or equivalently 

e(N ® M^" 1 ), V) = e(B) ■ co k/ko {a) = p M {ei). 
If this is satisfied, then by Theorem II 1.21 the theta lift is equal to the representation 
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of U(L a + L_i), with 

pN^ifi) = PnUi) ■ u k /k (-l) ■ e(Ni <g> Mi^ 1 ),^)- 

(b) Homu(L a )(vr PjvV /iz/, r]/r] cr ) ^ 0. This is a branching problem for U(2) x U(l) which we 
have resolved in §101 Using the results there, we see that the desired non-vanishing 
holds if and only if 

PNv(fi) = PN(fi) ■ Uk/kai-l) ■ e(Ni ® Mi^" 1 ),^) = e(Ni ® M^p" 1 ),^) 

or equivalently 

pjvC/i) = e(iV 1 (g, M(fj,~ 1 ),^ 2 ) = e(Nt ® M(/i~ 1 ), ^). 

Finally, since 

PA/(-l)=p J v(-l) = e( J B), 

we conclude that 

PA/(e 2 ) = e(AT®M 2 (p~ 1 ),V). 

Thus we conclude that 

Hom u(M / s) (tt Pm ® tt Pjv ® w^;, C) ^ 
if and only if pm x pAr is the distinguished character % of [GGP, Conjecture 16.3]. 

13. Restriction from U(3) to U(2): endoscopic case 

In this section, we consider the restriction problem for U(3) x U(2). Using theta 
correspondence, we establish [GGP, Conjecture 16.3] for endoscopic packets of U(3). 
In the following section, we shall consider the stable packets of U(3). 

We fix a pair 

W C V 

of split hermitian spaces of dimension 2 and 3 respectively. Without loss of generality, 
we assume that discK = 1, so that V/W is a rank 1 hermitian space L\ of discriminant 
1. Let W C V be hermitian spaces of dimension 2 and 3, such that V/W = V'/W. 

More concretely, for each quaternion algebra B over ko, we have a rank 2 hermitian 
space Vb- Then the rank 3 hermitian space 

V B , b = V B + L b 

has discriminant satisfying 

uJ k / ko (disc(V B ,b)) = e(B) ■ u k/ko (b). 
If we take 6 = 1, then as B varies, the pair 

V B C V b ,i 

gives the pairs W C V and W C V. 
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Suppose first that N is a 2- dimensional conjugate-symplectic representation of WD{k) 
with associated Vogan packet LTv of U(Vb). If N — ffijiVj, then we write 

A N = Y[A Ni = Y[Z/2ZU 

i i 

For the fixed given additive character ip of k/ko, we translate ip by —2 ■ disc(V) = —2 
and use the resulting character r/>_ 2 to fix the Vogan parameterization 

J(^_ 2 ) : U N < > ln{A N ). 

Now consider a 3-dimensional conjugate-orthogonal representation 

M = M 1 + M 2 

with dimMj = i and such that each Mj is conjugate-orthogonal. Unless, M = 3M%, 
we may further assume that Mi does not occur in M 2 . We shall assume that this is 
the case, since the other case is similarly handled. Then 

Am — Am x x A M2 

and we write: 

A Ml = Z/2Ze and A M2 = JJZ/2Ze, 

i 

if M 2 = ®iM 2>i . 

Moreover, we shall assume that the conjugate-orthogonal character Mi has a conjugate- 
symplectic square root. This can be achieved by twisting M, and since this twist can be 
absorbed into iV for the purpose of the restriction problem, there is no loss of generality 
in making this assumption on M\. Under this assumption on M%, we have described 
in §8] a construction of the Vogan packet I1m as well as a bijection 

n M < — ► Itt{a m ) 

which is canonical in this case (i.e. independent of the choice of any additive character). 
To recall the construction briefly, we set 

Mi = /x 2 

for some conjugate-symplectic character \i and set 

N' = M 2 • /I, 

so that N' is conjugate-symplectic and A^> = Am 2 - Then, for quaternion algebras B 
and B' over ko, one considers the theta correspondence for 

U(W B ') x U(Vb,i) 

relative to the data (ipo,-2,^), where ipo is our Axed additive character of ko- The 
packet ITm is then the theta lift of the packet II7V' of XJ(Wb>)- For the labelling of the 
representations in U M by Itt(A m ), we refer the reader to the end of §|8j 
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Now we would like to determine 

Hom u( y s) (7rM ® n N , C), 

for tvm € I1m and ttn G 11^. We examine this restriction problem using the see-saw 
diagram 

\J(V B + Li) \J(W B >) x \J(W B >) 



U(Vb) x U(Li) U(W^)- 

On U(Wb')) we start with a representation 7r^, E Hn' indexed by a character r] of 
Ajv', so that 

=£(£')■ 

On U(Vb), we start with a representation 7rY. associated to a character pjv of An, so 
that 

PJV (-1) = e(B). 

Then we have the see-saw identity: 

Hom u( v B) (e^ 0i _ 2)M (7r^,)(8)7r pjv ,C) = Hom u(w - B ,)(e^ i _ 2i ^ )VB>WB ,(7r^ v )(8)w^ )i _ 2)MiLl)WB ,, 
Note that 

with 

PmU n , =?7 and p M (e) = e(S') • ^(-1) = e(B) ■ e(B'). 
Moreover, 7rY. has Vogan parameter (relative to J(^ ,-2)) 

(iV v ,p Jv v) = (iV v ,p Jv ./3 ) 

with 

A)(/i) = ^/fe (-l). 

Then the see-saw identity reads: 

Hom u( y s) (7r PM <g> 7r PiV ,C) = Homu(w B o(^,-2,M 2 ,VB,w B ,(T / 9 Jv v) 
The RHS is nonzero if and only if (i) and (ii) below hold. 
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(i) 6'vo.-2,^ 2 ys,VK s /( 7r p Jv v) 7^ 0- By proposition lll.il this holds if and only if 

e(iV v /!- 2 ,<M = e(B) ■ e(B') = p M (e), 

or equivalent ly 

e(N®M 1 ,1>)=p M {e). 
Moreover, by Theorem lll.2[ when this holds, we have 

where p is the character of A N v = A N given by 

Po(fi) = e(iV^- 2 ,^-i) = <N t ® Af ls V). 

(ii) Homu( Ws , (vr PivV . po ® a;^ _ 2)M) jy B ,, 7r^,) 7^ 0. This question was addressed in the 
previous section, and we deduce that the desired non- vanishing holds if and only 
if the character 

(Pn ■ Po, v) e I rr (^v) x lrr(A N >) 

is the distinguished character xo m [GGP, Conjecture 16.3] for the skew-hermit ian 
case for (Wb',P>)- More precisely, the desired non- vanishing holds if and only if 

p N (fi) ■ e(N, <g> Mi, V) = eW ® (iV') v (/i)^-i) = e(iV< ® M 2 , 
so that 

p N (f l ) = e(N i ®M^), 

and 

fjfe) = e((iV;) v <g> iV v (//), V_i) = e(M 2 , ® JV, V)- 

This shows that 

Hom U (y B )(7Tp M (g) 7T pAr , C) 7^ 

if and only if the character pm x p N is the distinguished character xo of [GGP, Con- 
jecture 16.3], computed using the additive character ip of k/k . 

14. Restriction from U(3) to U(2): stable case 

We now consider the restriction problem for stable Vogan packets of U(3). We 
preserve the notation of the previous sections. In particular, we have the pairs of 
spaces W C V and W C V, with disc(V) = 1 = disc(V/W). Moreover, we use the 
additive character to normalize the Vogan parameterization for U(W). 

Let M be an irreducible 3-dimensional conjugate-orthogonal representation of WD(k), 
so that its associated Vogan packet has the form 
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where ttm is a representation of U(V) and n' M is the same representation considered 
on U(V'). If M is an irreducible representation of the Weil group W(k), then the 
representation n M is supercuspidal. Otherwise, 

M = fits S 3 

where /i is a conjugate-orthogonal character of W(k) and S 3 denotes the irreducible 
3-dimensional representation of SL 2 (C). In this case, the representation n M is a twisted 
Steinberg representation 

n M — St <g) (/i o det). 

On the other hand, let N be an arbitrary 2-dimensional conjugate-symplectic rep- 
resentation of WD(k) with associated Vogan packet II jy of U(W). We would like to 
determine 

Hom u(W )(vrM <S> rc N , C) 

for 7Tm £ IIm and itn £ Ily- We shall reduce this question to the case when Um 
and IIjv are both supercuspidal packets, by first treating the other cases directly. The 
supercuspidal case will then be handled by a global method. 

We first consider the case when M — /i Kl S 3 . Since we can absorb the twist by /i 
into the parameter N, we may assume without loss of generality that /i — 1. In this 
case, t\m — St is a quotient of a (un-normalized) principal series representation: 

0^1^ Ind^ (y) (l) — > St — > 0. 

We have: 

Proposition 14.1. //AT not the parameter of the Steinberg representation of 
XJ(W), we have 

Romu (w) (St <g> tt Pjv ,C) = Hom u(w) (7 s (l) ®7T PJV ,C) = Hom u(i) (7r Pjv , C). 
In particular, Hom U ( W /)(5't ® 7r Pjv , C) ^ and on fa/ i/ 

p N (f i ) = e(N i ,,{>) = e(N i ®M,,/>). 

If N is the parameter of the Steinberg representation of XJ(W), so that U N = 
{St V (w),l V (w>)}, we have 

Romu( W )(St <S> St V (w),C) = Hom U ( W )(/ B (l) <g> Stu( W ),C) ^ 0. 

On the other hand, 

Homu(w)(lu(v)) Q) + Homu(H")(5iu(v)i C) 

is eona/ to 

Hom u(W /)(Ind2 y lu(w)) = Hom u(z/) (1 U(H / , C) = C, 
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SO that 

Hom U ( W //)(S't U (v'), C) = 0. 

This proposition verifies [GGP, Conjecture 16.3] when M = /i®5 , 3 and TV is arbitrary. 
We may thus restrict attention to the case when M is an irreducible representation of 
W(k), so that Um is a stable supercuspidal packet. 

Next, we consider the case when 

jV = P+(P CT ) v or (jl®S 2 , 

where P and (P°") v are not necessarily distinct. In such cases, the associated represen- 
tations of U(W) are contained in principal series representations of U(W). Thus, we 
need to compute: 

Hom u(w) (vr M , Ind^ (W) (x) ) 

for a supercuspidal representation tim of U(V). By Frobenius reciprocity, we see that 
this is equal to 

Hom T ((7r M ) z ,x) 

where the unipotent radical Z of B C U(W) coincides with the center of the unipo- 
tent radical of a Borel subgroup of U(V). But (tim)z is isomorphic to the regular 
representation S(k x ) of T = k x . Thus we have 

Hom u(W )(7r M ,Ind^ (W/) (x)) = Hom fex {S(k x ), x) = C. 

This proposition is proved by a standard application of Mackey theory. Indeed, it is 
a special case of [GGP, Theorem 15.1], and so we omit its proof here. The main point 
to note is that the proposition verifies [GGP, Conjecture 16.3] when 

N = P + (P CT ) V with P ^ (P CT ) V , 

as the principal series representation on U(W) is irreducible. If P = (P CT ) V , the param- 
eter iV is dihedral with respect to k/ko and the corresponding principal series represen- 
tation of U(W) is the sum of two irreducible summands, and we have not determined 
which of these summands contribute to the 1-dimensional Horn space above. 

Finally, when N = /i <g> S 2 , we may assume without loss of generality that /i — 1 (by 
absorbing fi into M). Then 

TIn = {St V (w), lu(w)}> 

and 

-> lv(w) Ind^ (W/) l -> St v( w) — > 0. 
The above computation shows that 

Hom uw (vr M ,Ind^ (H/) ) = C. 
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On the other hand, by [GRS], we have 



Hom u(H /)(7rM,C) = 



and 



Hom u(W )(7r^,C) = 0. 



Indeed, if these Horn spaces were not zero, tim and n' M would be obtainable as a theta 
lifting from some U(2), contradicting the fact that M is a stable parameter of U(3). 
Thus, we conclude that 

Homu^) (ttm, Stu(w)) ^ 0, 
which is what [GGP, Conjecture 16.3] predicts. 

We are thus left with the case when M is an irreducible representation of W(k) and 
N is a supercuspidal parameter. For this remaining case, we shall prove the following 
local theorem by global methods. 

Theorem 14.2. Let W be a 2 dimensional hermitian subspace of a hermitian space 
V of dimension 3 over k. Suppose that n M (resp. n N ) is an irreducible supercuspidal 
representation of\J(V) (resp. XJ(W)) with irreducible Langlands parameter M (resp. 
N) ofW(k). Then e(M <g> N,ip) for a character ip of k/k is independent oft]), so may 
be denoted as e(M <g> N). Suppose that Homu(w/)(7rAf <E> ^n, C) ^ 0. Then 



Remark : The method that we follow to prove this theorem is pretty general, but it 
is based on a global theorem of Ginzburg, Jiang, and Rallis [GJR3, theorem 4.6] which 
assumes that automorphic forms on unitary groups U(n) have base change to GL(n) 
something which is known at the moment only for generic automorphic representations 
on quasi-split unitary groups. However, by Rogawski [Ro], base change is known for 
any unitary group in 3 variables, which is why we have restricted ourselves to U(3) in 
the above theorem. Nonetheless, we have formulated some of the preliminary results 
below in greater generality. 

We begin with the following globalization result about local fields, which will be 
applied to globalize hermitian spaces over local fields so that there is no ramifica- 
tion outside the place being considered, and the unitary groups at infinity are either 
compact, or of rank 1. 

Lemma 14.3. Let k be a quadratic extension of a non-archimedean local field ko- Then 
there exists a totally real number field F with k as its completion, and a quadratic 
totally imaginary extension E of F with corresponding completion k such that E is 
unramified over F at all finite places different from k. 

Proof: Except for the requirement about E being unramified at any place of F outside 
the place k, this is well-known. Suppose that a quadratic extension E 1 over i*\ with 



e(M <S> N) 
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possible ramifications is constructed. Then a well-known technique, crossing with a 
field, says that after a suitable base change, one can get rid of the ramifications; we 
leave the details to the reader. 

Lemma 14.4. Let W be a hermitian space over k. Let F be a totally real number 
field with completion k at a place of F, and let E be a quadratic totally imaginary 
extension of F with corresponding completion k. Then there is a hermitian space W 
over E giving rise to W over k in such a way that the corresponding unitary group 
is quasi-split at all finite places of F except the one corresponding to the completion 
ko; and at all but one infinite place the group is the compact group U(n), and at the 
remaining infinite place, the group is either U(n), or U(n — 1, 1); if n is odd, we can 
assume that the group is compact at all the infinite places. 

Proof : The proof of the lemma will depend on the well-known classification of a 
hermitian form over a number field, according to which a hermitian form over a number 
field is determined by 

(1) the normalized discriminant, and 

(2) the signatures at the infinite places. 

Moreover, given any normalized discriminant, and signatures at infinite places (except 
for obvious compatibility between normalized discriminant and signatures), there is a 
hermitian form. 

We also note the following exact sequence from classfield theory, 

-> F X /NE X -> A*/NA* -> G&L(E/F) -> 0, 

from which it follows that one can construct an element in F x which is trivial in 
F* /NE* at all the finite places except ko, and which at the infinite places has the 
desired signs, except that the product of the signs is 1 or -1, depending on whether the 
element in k^ /Nk x is trivial or nontrivial. 

The proof of the lemma is now completed by observing that a hermitian form of 
normalized discriminant 1 over a non-archimedean local field defines a quasi-split group, 
and that the normalized discriminants of the hermitian spaces Z 1 Z 1 + Z 2 Z 2 + - • - + Z n Z n , 
and Z 1 Z 1 + Z 2 Z 2 + • • • + Z n _\Z n _\ — Z n Z n over C are negative of each other, and if n is 
odd, the normalized discriminants of the hermitian spaces Z^Zx + Z 2 Z 2 + • • • + Z n Z n , 
and —{Z\Z\ + Z 2 Z 2 + • • • + Z n Z n ) are negative of each other. 

Corollary 14.5. Let W be a hermitian subspace of codimension 1 of a hermitian space 
V over k. Let F be a totally real number field with completion ko at a place of F, and 
let E be a quadratic totally imaginary extension of F with corresponding completion k. 
Then there is a hermitian subspace W of codimension 1 of a hermitian space V over E 
giving rise to W and V over k in such a way that the corresponding unitary groups are 
quasi-split at all the finite places of F except the one corresponding to the completion 
ko; assuming F ^ Q, the group U(V) is the compact group XJ(n + 1) at all but two 
infinite places, and at the remaining infinite places, the group is either U(n + 1), or 
U(n, 1); the subgroup U(W) is compact at all but possibly one infinite place. 
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Proof : Let V = W © L c where L c is k with the hermitian structure cZZ, c 6 fc x . 
Globalize W by the previous lemma, and globalize c so that the normalized discriminant 
of V is 1 at all the finite places of F other than k (so that U(V) is quasi-split at all 
the finite places of F other than ho), and so that c has arbitrary signs at infinity, with 
only the product of the signs pre-determined which allows for the desired conclusion. 

We omit a proof of the following corollary of the lemma which follows exactly as in 
the previous corollary. 

Corollary 14.6. If n = 1,2 mod 4, then hermitian spaces W C V of dimensions 
n,n + l can be globalized keeping them positive definite at infinity, and maximally split 
at all finite places other than k. For n = 2 mod 4, if W has an isotropic subspace 
of dimension n/2, and for n = 1 mod 4, if V has an isotropic subspace of dimension 
{n + l)/2 ; then there are an even number of real places in F , else an odd number of 
real places. 

Proof of Theorem 14.2: By the corollary above, we can assume that U(V) is compact 
at infinity. It is then easy to see that we can globalize the representation tcm of U(V) 
to an automorphic representation IT of U(V)(A) in such a way that it is unramified 
at all the finite places of F except k . 

By Lemma 1 of [P6], we can globalize n N to an automorphic representation Il such 
that the period integral 

/ /o/i ^ 0, 

^U(W)\U(W)(A) 

for some / in n , and f\ in LTx. 

By the theorems due to Ginzburg, Jiang, and Rallis, cf. [GJR3, theorem 4.6], since 
the period integral is nonzero, the central critical L-value, 

L(lx®nf)^o, 

where LT^ and LTf denote base change of LTo and LTi to E. 
This implies that the global root number, 

e (i,n £ ®nf) = i. 

Let 

n = ® w ii 0iW , and rii = ® w rii )tt) , 

with n 0i „ = 7T/v, and LT^,, = n M . From the nonvanishing of the period integral, it 
follows that 

Hom u(Wiu) (n , w ® n 1)tt) , C) ^ 

for all places w of F. 
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Since the representations LTi^ for w, a finite place of F, not v, are unramified by 
construction, they are in particular quotients of principal series representations, and 
hence we know the validity of theorem 14.2 for such representations: 



for all finite places w of F except v. 

Since the global epsilon factor is a product of local epsilon factors, we have 



The following lemmas then complete the proof of the theorem on noting that there 
are an even number of places at infinity if U(W) is quasi-split, and odd number of 
places at infinity when U(W) is not quasi-split. 

Lemma 14.7. Let W be a codimension 1 hermitian subspace of a positive definite 
hermitian space V of dimension n + 1 over C. Suppose that 7r (resp. t\\) is a finite 
dimensional irreducible representation of U(V) (resp. U(W)). Let the Langlands pa- 
rameter of ttq (resp. TTi) be a (resp. 0\). Suppose that HomumQ C^o <S> 7Ti,C) ^ 0. 



Proof: The proof of this lemma is a simple consequence of the well-known branching 
law from the compact group U(n + 1) to U(n), combined with the value of the epsilon 
factor given by the following lemma, which has been demonstrated in Proposition 12.11 

Lemma 14.8. Let ip be the additive character on C given by ip(z) = e~ 2my where 
z = x + iy. For n an integer, let \n denote the character Xn{ z ) = (z/z) n ^ 2 = e~ m9 for 
z = re w G C x . Then for n odd, 



Lemma 14.9. Let itq (resp. tti) be a finite dimensional irreducible representation of 
the compact group U(n) (resp. XJ(n + 1)) with L-parameter restricted to C x given by 
an n-tuple of half-integers a = {— A n < — A n _i < ■ • • < — Ai} (resp. a% = < fi2 < 
■ ■ ■ < /i rt +i} an (n + l)-tuple of half-integers), where all the X^s are half-integers but 
not integers if n is even, and are integers if n is odd, and /i-s are all integers if n is 
even, and half-integers but not integers if n is odd. Then Hom U („)(7T 1 <g> 7r ,C) ^ if 
and only if 



e ^(-,ng w ®rLfj = i 



e(-,M®JV)e 0O (-,IlJ oo (8)n? oo ) = l. 



Then 




if n = 0, 3 mod 4 
if n = 1, 2 mod 4. 




< Ai < /i 2 < A 2 < ■ ■ • < X n < Hn+i- 
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Corollary 14.10. With notation as in the lemma, and assuming that 7Tq appears in 

e(x Mfc ®<7o) = (-ir- fc+1 , for all k, 

and therefore, 

n+l 
k=l 



Remark : It should be mentioned that the global method followed in the proof of 
theorem 14.2 proves that if there is an invariant linear form, then the epsilon factor 
has some fixed value. The natural variant of the theorem of Waldspurger in [Wa4] to 
unitary groups proves that such an invariant form exists on a relevant pair of unitary 
groups, which will then strenghthen theorem 14.2 to an if and only if statement. 
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